
### Introduce yourself
I'm David. I'm a backend engineer. I use Go and Elixir as my main programming languages with a little bit of Javascript. My education and previous work experiences is related to GIS/cartography so I'm not what may be considered a traditional computer science person. But I have always been interested in tech. My curiousity always leads me to learn how and why things work. It is the reason why 5 years ago even while targeting a frontend role as a career shifter I got *"nerd sniped"* into learning Elixir.

### Why hire me?
Hungry, and excited but also pragmatic and experienced.

### What are your personal goals?
In 5-10 years, I want to be a staff engineer on product-infra/platform* in a "tech-native" company, or any kind of role that has direct impact to end-users whether they be external or internal. I also want to achieve financial stability, enough that I could choose the companies I work with and the roles I take. I have started working on this during this "not-by-choice" downtime, working on a personal curiousity venture I am hoping to turn later on to an actual business.

## Situational Questions

### Tell me about a time when you faced a challenging technical problem. How did you approach it, and what was the outcome?
There was a bug I was tasked to fix before I left to join a new company. Out of nowhere, the service would be restarted due to OOM. The OOM events did not correlate to a surge in traffic. I tried to reproduce the issue and running pprof using sampled requests that was running right before the crashes but to no avail. Nothing stood out in the memory usage.
As it seemed like I was not likely to solve it in the remaining time, I focused on documenting what I have learned (eg. the problem seems related to deserialization of responses to outbound requests) and what tests I have tried so the next person who will tackle it does not duplicate efforts.
I later learned from the engineer who solved the issue that he used a memory pool of response structs to mitigate garbage collection and memory pressure. And although he also didn't replicate the issue, my documented guess was useful enough to test out his solution and eliminate the problem.

### Describe a situation where you had to work under pressure to meet a tight deadline. How did you prioritize tasks and ensure timely delivery?
In a previous company, I was part of the team that was tasked to improve availability of our SMS infrastructure. We needed to implement integrations to 2 new providers and also add a static configuration API to easily do manual failovers. Even in scoping, we aggressively identified which tickets were absolutely necessary and which ones were nice-to-have's. Hence, ending up with static configuration and manual failover. This proved to be optimal as engineering was eventually outsourced but in the meantime I delivered better availability and improved costs (the new providers had much cheaper rates). Note also that the other engineer who was supposed to do the work with me was pulled to yet another time critical project. 

### Give an example of a time when you disagreed with a team member on a technical decision. How did you handle the situation, and what was the result?
This is a more common scenario than I'd like to be so I usually try to not think about it too much. If it's a matter of taste and we're on deadlock, I give way and implement/redo as advised so I can move on to other stuff. However, in cases where I will be operationally responsible for the feature/service (eg. support, on-call) I will consider advise from other people but do what I think is best.
For a specific example, in implementing the aforementioned new SMS integrations I decided to put all of the client configurations in a singleton hashmap that lived for the entirety of the service. Since this was only updated on redeploys, I thought it better to forego the multiple function calls that were simply looking up static values (as was done in the current provider implementation) and consolidate the configuration into a single place. To be honest, there was no noticeable performance benefit to doing this but visually it made the code a little more tractable as you only have to look at one place in the code. From what I recall, there was some slight objection to this from my teammates but since I ended up being the only one implementing it, it was ultimately my call.

### Tell me about a project where you had to learn a new technology or programming language quickly. How did you go about acquiring the necessary skills, and how successful were you in implementing the solution?
In two previous companies that I worked at, I had to learn a new language and get up to speed in a couple of months. The first one with Elixir and the second with Golang. Elixir was particularly hard because it was so different to the languages that I was familiar with. Plus, Phoenix the defacto web framework used a lot of DSLs powered by Elixir's metaprogramming which made it a lot weirder.

In both cases, when I'm blocked with something that I couldn't understand I search for Youtube videos to help me get a high-level view/explanation. Usually once I understand the concept, the syntax was easier to digest. If not, I copy an existing solution from our codebase and try to "retrofit" it to my own usecase using context clues. Installing the right tools like a linter or language server helps a bit here. Of course, I was also not familiar with these stuff before. I was very fortunate to land on teams with very helpful colleagues who are also good engineers and in almost all of my teams I was usually the least experienced and so I learned a lot from them.

## Give an example where you worked on something highly impactful
In a previous workplace, at least a couple of our external endpoints' require a currency exchange somewhere in their workflow. This was a "solved" problem but for some reason one time the internal service we have for this suddenly was not working. During the investigation, we discovered another flaw in the implementation that could have severely impacted our financials if an edge case scenario was triggered. So after the investigation, I was tasked with rewriting that service to be highly available and to fix the flaw in the previous implementation.
Aside from that, the aforementioned SMS integrations I implemented greatly improved availability as well as significantly cut costs (>10%).

### Give an example of a time when you had to work with a challenging or difficult team member. How did you handle the situation, and what strategies did you employ to maintain a productive working relationship?
To be honest I haven't had experience with a team member that was challenging or difficult _all_the_time_. Sometimes, it's just the situation they are in, and when I try to put myself in their situation, it gets a little easier not to take it personally. Also, there's a famous quote that says "Never attribute to malice that which can be adequately explained by neglect" so if that's the case then I just have factor it in in my working dynamic with that person. Otherwise, I don't have to like other people to be able to work with them. I know how to be a professional.

### Describe a situation where you had to make a trade-off between implementing a feature quickly or ensuring its quality and stability. How did you approach the decision-making process?
One thing I learned the hard way is not knowing/establishing what an "exit strategy" looks like. Others may call it MVP but I think Exit is more apt as there is a notion of personal relation to the product/project. So in every phase of a project/product you have to be aware of that because it's nice to have well delineated timelines/deliverables but it's also unrealistic to expect it to not move, maybe people really liked the feature but it's missing just one little thing and you have to deliver it.
Fortunately I haven't been in a situation where a feature is absolutely required in less time than a sprint. So for better or worse, I have always leaned towards stability if not quality in the features or services I have built. But if worse comes to worst, I will implement as quickly as possible without sacrificing logging. I think that _emitting informative, structured logs should be the absolute minimum_. Everything else can be improved later; even tests.

### Tell me about a time when you made a mistake or encountered a failure in a technical project. How did you handle it, and what did you learn from the experience?
I alluded to it in the previous question. During the time when management started letting people go, I was tasked to investigate what changes are needed (if any) for a partner's compliance certification. Due to the partner not being able to provide the specific requirements they needed from us, it took us _A LOT_ of back and forth. And as I mentioned, I failed to establish my "exit" so I ended up taking on the point person role and I would dare say even the project manager role for this task. The changes required involved web, mobile and backend components. I escalated the situation to my manager, to other leads, to product and even to the projects head. Thankfully the mobile lead who was very proactive and a couple of people in web team helped me a lot on their end. We were able to push it to completion but on a delayed timeline.

Although it was a time of personal difficulty, I later learned that the delay was actually beneficial* to the company as it allowed company management to prolong the relationship with the partner until they were sure they were going to cut it off which they did a month or so after we delivered the project.

### Give an example of a time when you had to work on multiple projects simultaneously. How did you manage your time and prioritize tasks effectively?
IF I squint a bit, I kind of do this all the time. I usually don't do one task exclusively because tasks regularly get blocked on something and it's not always up to me and I have no control over it. So I take 2-3 small tasks that I can make small progress on without sacrificing the original task's timeline. Otherwise if I get too swamped and something's already missed the deadline, I complete the other tasks that are still "in the green". There is no point sacrificing other tasks timelines to make an already late deliverable just a little less later.

## Give an example of a project you were involved in where you were given not much guidance
I think apart from the compliance certification I have mentioned above, I don't have experience yet of a project where I was totally alone. In the companies and teams I worked with, I always found leaders and even colleagues to be helpful. Although their help is not always at "arms length" or tactical, ie. specific, I could rely on someone's guidance when I need it.

## What are some examples of companies you want to work for? and why?
1. **Amazon**. They're THE infrastructure company. I think every software engineer's (or product person's) aspiration is to leverage technology to benefit the most number of people. It is the same for me.
2. **Stripe**. They are also an "infrastructure company". Their informal mission/vision is *to increase the GDP of the internet*. I'd like to be part of that.
3. **Fly.io**. Same as AWS but for a slightly different customer base, and scale.
4. **Remote**. I think remote work is the future. The internet and the COVID pandemic have greatly nudged the world towards that future. But there is still a lot of work to be done for opportunity to be more equitably distributed. And I want to take part in it. I've actually interviewed with them twice now (as of Aug 15, 2023). The first time even before their series A funding round.

## Questions for the interviewer
### Recruiter
1. What does the organization structure look like? Which team do I join in case I get hired? Who will I report to? Which teams will I work with?
2. What are the next steps? How long does the whole application process take?
3. What is the salary budget for the role?
4. If I want to test some tooling, do I need to obtain some special permission to deploy it to test/stage?
5. What part of your work do you find the most interesting here at <company>?
### Technical
1. What's the onboarding process like? Where would a newbie look to debug an issue? What tools to support learning/debugging the system?
2. What are some ways ownership of code is encouraged/reinforced, eg. on-call?
3. What is the level of involvement devs have in infra? Is there an internal platform or do devs use AWS/GCP directly?
4. What's one interesting thing you've built in Elixir (or Go) here? How does being written in Elixir help, if at all?
### Miscellaneous
1. Is the team that I am prospectively joining an established one or just new? How many members currently?
2. What are some example of deliverables for the role?
3. Since the company is distributed, is there a required time overlap among teams?

## System Design guide questions
1. Time frame for exploring solution
2. Time frame for building solution
3. Time frame for maintaining solution
4. Who are the customers? stakeholders?
5. Projected scale
6. What are the moving parts? which parts are transactional?
7. What/where are the sources of truth? need to minimize these
8. Other constraints


Some questions I should probably write the answer to because they keep getting asked and I keep forgetting stuff at time of interview

Frequently Asked Interview Questions


Microservices are easy but let's make it even easier to write them.

1. Observability? Logging and metrics are good enough.
2. Backpressure? Async all the way.
3. Docs? Good code is self documenting.
4. Edge case? Just write another microservice and while you're at it make it handle other concerns the other implementations do not.
5. End-to-end tests? They're not as beneficial as you think they are.

Bonus: Remove internal platform/ops team from any discussion so there is less people considering additional burden/complexity in operating a separate service.

In case anyone think otherwise, **this is sarcasm**. Microservices are a *zero interest rate phenomenon*. Similar to writing small functions, writing microservices only looks "cleaner" at *write time* when you've just extracted logic from a monolith and all that context is fresh to you as the developer. The moment you're hunting down bugs and trying to piece context from a hundred components, you will be thinking otherwise. And if you don't have distributed tracing to help you, good luck.

Write-optimized microservices


Say a user wants to access an endpoint. Before their request is served by the upstream service, the system needs to validate that the user is who they say they are and to determine the authority they have. Authentication and authorization may differ depending on whether the user is a customer, an external partner or an admin staff. But although the paths might be different, this can be handled by a single service/component, or maybe another providing a slightly different internal API. The point is that they are logically interchangeable and the request paths do not totally diverge. Requests may fork at specific method or middleware (or even service) but they eventually "join" before/as they hit the upstream service. Another way to think about this is that the paths are confined to the same "domain component".

Let's generalize this a bit more. In the diagram below, `Service 1` and `Service 2` are distinct services but they both satisfy and are _constrained_ by the domain/business function that `Component A` serves. Neither `Service 1` nor `Service 2` take on any function of `Component B` to bypass it and directly go to `Component C`. Service to service hops are predictable. This make it easier for someone unfamiliar to the system to quickly understand the overall architecture as they are not pulled into rabbitholes of edge cases.
```ascii
            ┌───────────────┐       ┌───────────────┐       ┌───────────────────┐
    ──────► │ Component A   │       │ Component B   │       │ Component C       │
Client      │               │       │               │       │                   │
requests    │   Service 1   │       │   Service 3   │       │  Service 5        │
    ──────► │   ┌───────┐   │       │   ┌───────┐   │       │   ┌───────────┐   │
            │   |       |──►│       │   |       |──►│       │   │           │   │
            │   └───────┘   │──────►│   └───────┘   │──────►│   │           │   │
    ──────► │   Service 2   │       │   Service 4   │       │   │           │   │
            │   ┌───────┐   │       │   ┌───────┐   │       │   │           │   │
    ──────► │   |       |──►│       │   |       |──►│       │   └───────────┘   │
            │   └───────┘   │       │   └───────┘   │       │                   │
            └───────────────┘       └───────────────┘       └───────────────────┘
```

In functional programming, Scott Wlaschin describes this as [Railway Oriented Programming](https://fsharpforfunandprofit.com/rop/) where the success and the error cases follow the same path. On the OOP side, there are the classic design patterns which in their simplest essence are simply means to reduce interaction between similar function components to a single interface. And in Go they say "the happy path is left-aligned" and tips like "avoid excessive nesting".

In all of these, there is an essence of flow. In code/instruction level, the problem of breaking flow is called _cyclomatic complexity_. We want to avoid breaking flow because operators want systems that are easy to maintain and it is hard to maintain a system when one can only build a mental model of a small portion of it at a time because it is full of edge cases handled differently. There are build tools/libraries to measure this but their purview do not cross service boundaries and so, like state, it is up to engineers and operators to manage this implosion. And so we must.


The zen of programming and why protect it

A happy line


> AKA how I whittled days away trying to write a custom stream decoder
  
## `v0.1` *this is a Go program, so I must use channels. right?*

This hacky solution has a BIG problem. It doesn't know how much to read
from the buffer. My colleague has yet to introduce me to the idea of
using a bufio.Scanner/Reader.

```golang
func main() {
    decs, err := zstd.NewReader(rdr)
    if err != nil {
        logger.Fatal().Err(err).Msg("zstd.NewReader fail")
    }

    buf := new(bytes.Buffer)
    // pick random size large enough to fit a few screens of text
    buf.Grow(500000)

    brdr := bufio.NewReader(buf)
    done := make(chan struct{})
    work := make(chan struct{}, 20)
    // decompress stream to buffer since zstd does not provide an API
    // to read data until a delimiter param
    go func(zst *zstd.Decoder, buff *bytes.Buffer, stop, wrk chan struct{}) {
        // since the "row" length is variable, I used an estimate
        // and a buffer to buffer multiple rows
        bb := make([]byte, 100000)
        for {
            _, err := zst.Read(bb)
            if err != nil {
                if err == io.EOF {
                    break
                }

                log.Err(err).Msg("zst.Read fail")
                return
            }

            _, err = buff.Write(bb)
            if err != nil {
                log.Err(err).Msg("buff.Write fail")
                return
            }

            wrk <- struct{}{}
        }
        done <- struct{}{}
    }(decs, buf, done, work)

    fooCha := make(chan Foo, 20)
    // receive and batch process for insert
    go func(fooCh chan Foo, repo bar.Repository) {
        bars := make([]Bar, 0, 20)
        for fo := range fooCh {
            var b bar.Bar
            if !ConvertToBar(&fo, &b) {
                continue
            }
            bars = append(bars, sp)
            if len(bars) < batchSize {
                continue
            }

            copies := make([]Bar, batchSize)
            copy(copies, bars)
            // reset bars for next batch
            bars = bars[:0]

            if err := repo.insertBars(copies); err != nil {
                break
            }
        }

        // execute last batch
        if len(bars) != 0 {
            _ = repo.insertBars(bars)
        }
    }(fooCha, repo, o)

    // read from buffer and send to foo channel
READBUFIO:
    for {
        var bs []byte
        select {
        case <-done:
            // drain buffered lines after receiving done signal
            for {
                if bs, err = brdr.ReadBytes('\n'); err != nil {
                    if err == io.EOF {
                        close(propCha)
                        break READBUFIO
                    }

                    log.Fatal().Err(err).Msg("brdr.ReadBytes fail")
                }

                var f Foo
                err = json.Unmarshal(bs, &f)
                if err != nil {
                    log.Err(err).Msg("decode dump: json.Unmarshal")
                    continue
                }

                fooCha <- f
            }
        case <-work:
            // this case is to avoid reading into the buffer
            // before any data has been written;
            // the io.EOF block below in theory *should* never happen
            if bs, err = brdr.ReadBytes('\n'); err != nil {
                if err == io.EOF {
                    break
                }

                continue
            }

            var f Foo
            err = json.Unmarshal(bs, &f)
            if err != nil {
                log.Err(err).Msg("decode dump: json.Unmarshal")
                continue
            }

            fooCha <- f
        }
    }
}
```


<br />

## `v0.2` *where I learned how to count*

```golang
// this implementation is plenty fast BUT it uses as much
// memory as the size of the compressed zst file
func main() {
    zsr, err := zstd.NewReader(resp.Body)
    if err != nil {
        return err
    }
    scanr := bufio.NewScanner(zsr)
    buf := new(bytes.Buffer)
    buf.Grow(100000)

    for {
        // not sure why i even used a buffer at this point
        // instead of simply unmarshalling scanned lines
        count, err := bufferJSONLines(scanr, buf)
        if err != nil && count == 0 {
            if err == io.EOF {
                break
            }

            continue
        }

        // only decode as much lines as buffered
        foos, err := decodeFromBuffer(buf, count)
        if err != nil {
            continue
        }

        if err = repo.Insert(foos); err != nil {
            continue
        }
	}
}

func bufferJSONLines(scanr *bufio.Scanner, buf *bytes.Buffer) (int, error) (
    var scanned int
    for {
        if !scanr.Scan() {
            if err := scanr.Err(); err == nil {
                // scanner does not return io.EOF
                break
            }

            continue
        }

        if err := buf.Write(scanr.Bytes()); err != nil {
            continue
        }
        scanned++
    }

    return scanned, nil
)
```

## `v0.3` *Well f\* me. Why didn't you say so earlier?*

> type **Decoder**  
> &nbsp;&nbsp;A Decoder reads and decodes JSON values from an input stream. 
>> type Decoder struct {  
>> &nbsp;&nbsp;&nbsp;&nbsp;// contains filtered or unexported fields
>> }

Yes. Apparently.

```golang
type decoder struct {
    Decoder *json.Decoder
    BatchSize int
}

func (dc *decoder) DecodeBatch() ([]Foo, error) {
    properties := make([]domain.Property, 0, dc.BatchSize)
    var count int
    for count < dc.BatchSize {
        var pr domain.Property
        if err := dc.Decoder.Decode(&pr); err != nil {
            if err == io.EOF {
                return properties, io.EOF
            }

            if count == 0 {
                return nil, err
            }
            continue
        }

        properties = append(properties, pr)
        count++
    }

    return properties, nil
}

func main() {
    // ...
    zsr, err := zstd.NewReader(resp.Body)
    if err != nil {
        return err
    }
    dc := json.NewDecoder(zsr)
    dcodr = &decoder{
        Decoder: dc,
    }

    for {
        foos, err := dcodr.DecodeBatch()
        if err != nil {
            if err != io.EOF {
                continue
            }

            if len(foos) == 0 {
                break
            }
        }

        if err = repo.Insert(foos); err != nil {
            continue
        }
    }
}
```


How I whittled days away trying to write a custom stream decoder

JSON decode a TSV stream


At $WORK, I recently needed to make a change which could have been an easy change if only our system was designed differently. The difficulty boils down to one issue: the cache was "too far" from the pertinent data source.

Clients of cached components/services do not need to know about caching. They could care less if it's making a very expensive call to a third party service the other side of the world or simply fetching a response from memory. This is a property and the main selling point of interfaces.

## Unidirectional data flow

You can make all sorts of optimization between a source and its cache because these details are hidden behind an interface. On the other hand, the farther away (in layers of abstraction) a cache is from its source, the more complexity it eventually (it has to) leaks to the whole system.

Consider `System C` which depends on `Component A` and `Component B`. The two components share the same interface but have different implementations. `System C` is not aware that the two components have their own cache. From `System C` perspective, how the sources interact with their own cache (sca, scb) or even whether the caches exist or not does not matter.

```ascii
┌──────────────────┐
│      Component A │
│  ┌─────────┐     │◄─────────┐
│  │ Source  │     │          │
│  └─┬───────┤ sca │          │
│    ├───────┴─┐   │          │
│    │  Cache  │   │       ┌──┴──────────┐
│    └─────────┘   │       │             │
│                  │       │             │
└──────────────────┘       │             │
                           │             │
                           │  System C   │
                           │             │
┌──────────────────┐       │             │
│      Component B │       │             │
│  ┌─────────┐     │       │             │
│  │ Source  │     │       └──┬──────────┘
│  └─┬───────┤ scb │          │
│    ├───────┴─┐   │          │
│    │  Cache  │   │◄─────────┘
│    └─────────┘   │
│                  │
└──────────────────┘
```

Let's then say that as service owners of `System C` we decided we would like it to have its own cache to avoid calling the components, whenever possible. By doing that, however, we would be trading off clear separation of concerns for a few 10/100/ms of latency. `System C` would have to provide an API (sca', scb') for each of the components to invalidate/update the cache similar to how the component communicates with its cache internally.

```ascii
┌──────────────────┐
│      Component A │
│  ┌─────────┐     │◄─────────┐
│  │ Source  │     │          │
│  └─────────┘     │          │
│                  │          │
└──────────┬───────┘   ┌───┬──┴──────────┐
           │           │ c │             │
      sca' └──────────►│   │             │
                       │ a │             │
                       │   │             │
      scb' ┌──────────►│ c │  System C   │
           │           │   │             │
┌──────────┴───────┐   │ h │             │
│      Component B │   │   │             │
│  ┌─────────┐     │   │ e │             │
│  │ Source  │     │   └───┴──┬──────────┘
│  └─┬───────┤ scb │          │
│    ├───────┴─┐   │          │
│    │  Cache  │   │◄─────────┘
│    └─────────┘   │
│                  │
└──────────────────┘
```

Notice that when the cache was colocated with the source and exposed via the same interface to the "outside world", the data flow across components was unidirectional. There were no adhoc communication channels. It would be easy to dismiss this difference as trivial but add in another layer of abstraction between `System C` and the source components and the data flows easily gets hairy.

*I use the term here loosely to mean a single line of communication, and do not correspond to common usage e.g. React/Elm data flow.


Why hiding behind the interface is a good idea

Cache near your source


Writing down just recent ones from memory for now. This is a growing list.

## Personal
- [Dan Luu](https://danluu.com/)
- [Will Larson](https://lethain.com/)
- [the morning paper](https://blog.acolyer.org/) by Adrian Colyer. ELI20 for technical papers
- [StaffEng](https://staffeng.com/) also by Will Larson
- [brandur.org](https://brandur.org/articles) by some guy previously from Stripe
- [Nelson Elhage](https://blog.nelhage.com/post/) another guy previously from Stripe :D
- [Eugene Wei](https://www.eugenewei.com/) stuff on product/strategy
- [High Scalability](http://highscalability.com/)
- [Simon Eskildsen](https://sirupsen.com/) of Shopify fame
- [fasterthanli.me](https://fasterthanli.me/) if you like Rust

## Corporate
- [AWS Builders Library](https://aws.amazon.com/builders-library/)
- [Dropbox](https://dropbox.tech/)
- [fly.io](https://fly.io/blog/)
- [Shopify](https://shopify.engineering/topics/infrastructure)
- [Stripe](https://stripe.com/blog/engineering)
- [Uber](https://eng.uber.com/)

## Specific articles

- [The Dark Resistance](https://dr-knz.net/dark-resistance.html)
- [Invisible Asymptotes](https://www.eugenewei.com/blog/2018/5/21/invisible-asymptotes)
- [Shitlists](https://sirupsen.com/shitlists/)
- [Choose Boring Technology](https://mcfunley.com/choose-boring-technology)
- [APIs as infrastructure: future proofing Stripe with versioning](https://stripe.com/blog/api-versioning)
- [Stripe's payment APIs: the first 10 years](https://stripe.com/blog/payment-api-design)

Awesome tech blogs


A few weeks ago at work we launched a new microservice which I had the opportunity to solely implement. Aside from some deserialization issues (an external API dependency with inconsistently typed response fields), everything seemed to be OK. Then it wasn't. Our alerting system in GCP (Google Cloud) was on fire only hours after we went live.

The pods were on a crash loop. And there were no logged errors on Stackdriver.

I added all sorts of error logging and a few `recover()` in critical places. (Yes. This would probably be less painful if we had distributed tracing in place. We make do.) But still, there was nothing. There was something curious though, the `info` logs for inbound requests did not match the outbound requests. The crash involved only one endpoint that accepts an array, among other fields, and for each of the items in the array, the service makes one external call. Initially, I attributed this to the crash dropping the other calls and assumed it to also be the case for the "missing" `error` logs. I eventually learned this assumption to be incorrect. There were really no error logs because there were no errors.

We found a hint in the `events` tab of the corresponding Kubernetes workload in GCP console. The workload would fail a `healthcheck`probe before a "crash". This was tricky to catch because Kubernetes events are stored in the apiserver and by default are not persisted and kept only for an hour. You might have already noticed I enclosed "crash" in double quotes. We didn't immediately realize what was happening to the pods. I had correctly implemented and exposed the healthcheck endpoint. We could call it just fine. We left it as is for a day or two since the crashes were not really business critical and we were also out of ideas.

Then we discovered the latency problem. On requests with very big arrays previously mentioned, the latency spikes were much higher than other services usual response times. This led me to another discovery. The response time for the healthcheck endpoint would range north of 1s when the request is made concurrent to a "big array" request. A usual response time for a healthcheck probe would only range in the order of 10ms to 100ms. In short, "big array" requests were starving healthcheck probes. This was the lightbulb moment. So I asked what are default GKE/Kubernetes timeout config, if any. I then learned that in Kubernetes by default workloads have a healthcheck probe timeout of 1s and a retry of 3. Failing this, the scheduler would garbage collect the "unhealthy" pod and deploy a fresh instance. Thus the crash loop.

Bumping resource limits and timeout up a few solved this.

The case of the crash looping pod


1. Create separate SSH keys and add them to appropriate Github account settings. Make sure they are easily distinguishable from each other.

2. Modify the ssh config file ( ~/.ssh/config) or create it if there isn't one.

```
# Personal GitHub account
Host github.com
 HostName github.com
 User git
 AddKeysToAgent yes
 UseKeychain yes
 IdentityFile ~/.ssh/id_rsa

# Work/other GitHub account 
Host workhub
 HostName github.com
 User git
 AddKeysToAgent yes
 UseKeychain yes
 IdentityFile ~/.ssh/work_rsa
```

The `host` alias defined in previous step may now be used to manage repos from both accounts. 
e.g.
git URL without the alias: `git@github.com:<organization>/<project>.git`
with it: `git@workhub:<organization>/<project>.git`
```
$ git clone git@workhub:<organization>/<project>.git
```

Extra: Git can be configured to also swap URLs so that the alias does not need to be specified when
cloning/adding new repositories.
e.g. `~/.gitconfig`
```
[user]
    name = David Ebreo
    email = david@<organization>.com
[core]
    editor = vim
[alias]
    co = checkout
[url "ssh://git@workhub/<organization>"]
    insteadOf = git@github.com/<organization>

```  

Hope that's useful and clear enough. There are more detailed (and probably clearer) instructions [out there](https://xiaolishen.medium.com/use-multiple-ssh-keys-for-different-github-accounts-on-the-same-computer-7d7103ca8693). This here is mainly to jog my memmory when I forget how to do it. 

Just a personal note to jog my memory from time to time

Using Multiple SSH Keys for Github


- [x] Primitive types
- [x] Error Handling
- [x] Dependency management
- [ ] Async and/or Concurrency
- [ ] Abstraction
- [ ] Tooling
- [ ] Testing

Great! You want to learn a new language. You're bored at work and you realize you're current workhorse is limiting, or the stack at your new gig uses this weird esoteric stuff with too many parentheses and you hope you could understand even just some part of it. Either way, you want to learn one as efficiently as possible coz you've got other stuff to do. Who learns them for fun anyway? You a nerd? :p

Read the defacto "bible" of the language. A good rule of thumb for page count should be around 300-500 pages. Anything more is just too much detail that can be easily googled anyway. Start with the table of contents. DO NOT skip it. A good book's table of contents gives readers an idea what the whole book is about. For programming language textbooks, the table of contents summarizes the language API. That's a major productivity win. You get the gist of your new language powers in less than 10 pages.

The checklist above is one I am using in learning Go. It's even shorter than a table of contents. Just learn enough about each item to distinguish differences in semantics (if any) between the language you're learning and any you already know. Do now worry about syntax. It will come with use. Once you have a basic understanding of the similarities/difference, move on to a different item. Try it and let me know if it works for you.

Learning the ways a new programming language does stuff

New Language Checklist


Developers are obsessed with tools. It's our natural tendency. The capacity for toolmaking separates us from our less evolved counterparts in the biology tree -- the ability to build on top of the existing, to refine the crude. It's only too easy to get too involved in the tools than in the "building up" that they should be used for.

This past year I'm glad to have reached a certain level of comfort with developer tooling that I was able to invest (I should say waste) my time learning other programming concepts, patterns and techniques. At work, we use the Elixir programming language for our backend services and while compared to other ecosystems, Javascript in particular, the leadership and community of Elixir takes a more opinionated approach to tooling, this did not prevent me from spending too much of my time "researching" about the hottest languages, frameworks, etc. In other words, yet more tooling. To be fair, it was my curiousity and chasing that rabbithole that led me to become a tech professional. It was only around a month earlier this time last year when I joined my current company as a junior web developer. It was luck not mastery. I was fortunate to get employed even with such haphazard means of learning. Moving forward, I need to be more disciplined and focused in the use of my time.

Tooling and technology in general is an implementation detail. As developers, our ultimate goal is to create and expose value. If we are successful enough, the details don't matter. We do not obscure value with detail. This should be our objective as technologists, to shave away as much detail as possible. And yet many times, it is the opposite that rings true. Recent trends bring this problem to attention.

For much of the hype the blockchain community has produced, a community I belong to only in periphery (our company develops blockchain connected products), it has yet to deliver any significant value for people outside its bubble. Most of the debates and discussions surrounding it revolve only around nuances of implementation. For blockchain to succeed, it needs to kill its own hype and focus its collective attention and brainpower on the massive [foundational work](https://threadreaderapp.com/thread/1078682801954799617.html) that yet lay ahead. People should not forget blockchain technology, as cool as it is, is simply a tool like all other technology. Its value proposition is decentralized trust and security but they are not the goal. End-user apps should be able to build on top of those fundamentals.

A few days ago, in a tech slack I was a member in, someone asked the group for advice on whether to pursue a career in game development or continue as a web developer as he feels he has already plateaued in his career and have recently become complacent. I had a sudden realization -- people have the wrong (personal value judgement) mindset about technology. We end up building careers around technology because we optimize for mastery. It's not a bad aspiration, it's actually noble but that's not the point. But why did that guy feel so uninspired? What made him that way? My hunch, he lost touch with the value he built, and the people he built it for. Technology is so big in breadth and in depth, this is especially true for newcomers like myself, that simply keeping up with the trends is a feat in itself. Developers can build up careers by scaling up and down the slope of these tech trends but sooner or later that drive dies down. There is only so much meaning to be derived from following, learning, and conquering trends. It must be done for something, for someone. We need to be able to connect to that. If we cannot see or feel the value in what we create with the work that we do and the tools (however shiny they are) that we use, or are unable to share it with other people, we are wasting our time.

This coming year my personal aim is to focus on value-creation (or product) driven learning. This means I will consciously avoid pursuing "bling" technology for the sake of curiousity. That also means, for the next few months I will need to channel my learning on software delivery in the tech stack I am already comfortable in. With that said and out of the way, happy shiny new year everyone!

Shiny Shiny


> Explicit is better than implicit. 
> *Zen of Python*

In my previous job, writing scripts to automate some of the repetitive tasks got me back into the programming route which eventually led me to consider a [career shift](/about). To be honest, automating tasks was, at least upfront, more hard work than additional straight up keystrokes. And without guarantee that the time invested will return in any quality or amount of savings, especially for smaller tasks, it was a day's work worth of risk. But it was fun and for most of the time it got the job done. It was also a lot of hacking. In hindsight I realize, nothing of what I created that time was reusable. Many were just one-off scripts that I had to manually edit as often as I run them because I had hardcoded variable information instead of exposing them properly as parameters.

Any good software is only as good as its interface. For users, it's UI and the "frontend"; for programmers, it's parameters. Functions and their parameters are what technology people mean when they say "API". But what about parameters make them any good? Let's imagine a `list_activities` function.

```elixir
def list_activities(params \\ []) do
  sort_params =
    params
    |> Enum.filter(fn {k, v} -> k in [:asc, :desc] end)
    |> case do
      [] -> [:desc, :id] # default case
      order -> order
    end

  Activities
  |> order_by(sort_params)
  |> other_query_fn(params)
  |> Repo.paginate()
end
```

### Options

Good parameters are parameters you don't need to write. In our example, the function is defined with an optional keyword list param. But specific activities may be filtered by defining the param keys `:asc` and `desc` such that any of the following calls are also valid

```elixir
list_activities(asc: :inserted_at)
```
```elixir
list_activities(desc: :id)
```
```elixir
list_activities(asc: :inserted_at, desc: :verified_at)
```
In elixir,
`[asc: :inserted_at, desc: :verified_at]`
and
`[{:asc, :inserted_at}, {:desc, :verified_at}]`
are equivalent and do not need the enclosing list brackets when passed as the last param in a function

This way the caller is prevented from reaching for parameter details and provide them only as needed. As a rule of thumb, you may make default any parameter that covers the most common case for your application.

## Names

Good parameters are called by their name. If possible, meanings should depend only on their names by calling their specific keys rather than by the order they are called.

```elixir
def list_activities(t \\ :sports, order \\ :asc, field \\ :inserted_at)
```

could easily be mistaken with this and possibly result in a runtime error

```elixir
def list_activities(t \\ :sports, field \\ :inserted_at, order \\ :asc)
```

Do something like this instead

```elixir
def list_activities(t \\ :sports, opts \\ [field: :inserted_at, order: :asc])
```
Here we allowed the activity type to take on the position of the first param. For some, this might be still ambiguous but I think this is intuitive enough. The semantics here can be described and remembered as `list activities [where [how]]`.