For example, imagine a function that does not use dependency injection:

import config from './config'
import knex from 'knex'

async function saveUserPreferences(userPrefs: UserPreferences): Promise<void> {
    // Ok, so how do I get the DB?  Maybe I should just instantiate it!
    const db = knex(config.db)

    await db('user_preferences')
        .update(userPrefs)
        .where({ userId: userPrefs.userId })

    db.close()
}

The obvious flaw in this example is that I have to instantiate the db in every function that wants to make the call. Alternatively, if you're going to use a database connection pool, it becomes impossible since every repository function has its own instance. Another issue is that we have to manage the database client's lifecycle (by calling close at the end).

Of course, we could have magically imported the database from another module:

import db from './db'

export async function saveUserPreferences(userPrefs: UserPreferences): Promise<void> {
    await db('user_preferences')
        .update(userPrefs)
        .where({ userId: userPrefs.userId })
}

Now think about how we test this code. What if I want to verify that the database is called with the correct signature?

import { saveUserPreferences } from './userPreferencesRepository'

describe('saveUserPreferences', () => {

    it('should update preferences for the record with matching userId', async () => {
      // hmmm... how do I do this?  Do I need to run the DB locally?
      // Do I execute the update and then call the database to see if the change was successful?
    });
})

In the Jest unit test framework, you can override a module using jest.mock('./db'). You would then create a matching module in a relative folder called __mocks__ (e.g., __mocks__/db.ts) with your test implementation. However, this is not true for every language, and frankly, this pattern is ugly (sorry, Jest devs).

Inversion of Control

A much simpler alternative is to invert the component responsible for controlling dependencies to the caller (Inversion of Control). Instead of saveUserPreferences resolving or instantiating db on its own, what if we passed the db to the function?

import{ Knex } from 'knex'

export async function saveUserPreferences(db: Knex, userPrefs: UserPreferences): Promise<void> {
    await db('user_preferences')
        .update(userPrefs)
        .where({ userId: userPrefs.userId })
}

Unit testing becomes a lot easier:

import{ Knex } from 'knex'
import { saveUserPreferences } from './userPreferencesRepository'

describe('saveUserPreferences', () => {

    let db: (table: string) => Knex
    let ctx: Knex

    beforeEach(() => {
       // We are going to use a trick and partially implement the db.
       // Also, I probably have the type signatures of Knex incorrect.
       const ctx = {
            update: jest.fn(() => ctx),
            where: jest.fn() => Promise.resolve()>,
       } as unknown as Knex
        db = jest.fn(() => ctx)
    })

    it('should update preferences for the record with matching userId', async () => {
      const prefs = {
        userId: 42,
       email: 'foo@bar.com',
        phone: '555-867-5309',
      }
      await saveUserPreferences(db, prefs)
      expect(db).toHaveBeenCalledWith('user_preferences')
      expect(ctx.update).toHaveBeenCalledWith(prefs)
      expect(ctx,where).toHaveBeenCalledWith({ userId: prefs.userId })
    });
})

Of course, passing around db to every repository function will become a massive pain in the butt as the code base grows. More importantly, it makes callers conscious that saveUserPreferences uses some RDBS they probably shouldn't know about. In fact, our domain code should be relying on abstractions and not have any reference to Knex at all! How can we use DI but hide implementation details from the caller?

Using Factories

The answer is that you need a little bit of application code to hide the injection details. A straightforward way of doing this is to wrap your saveUserPreferences in a factory function and take advantage of closure scope, allowing the function to reference db without the caller knowing:

import{ Knex } from 'knex'

export function saveUserPreferencesFactory(db: Knex) {
    return async (userPrefs: UserPreferences): Promise<void> => {
        await db('user_preferences')
            .update(userPrefs)
            .where({ userId: userPrefs.userId })
    }
}

Now, when your application starts, you can build an instance of the saveUserPreferences function and pass around the built instance, which has a reference to the db in the upper scope:

const saveUserPreferences = saveUserPreferencesFactory(db);

await saveUserPreferences({
  userId: 42,
  email: 'foo@bar.com',
  phone: '555-867-5309',
})

Structuring Dependency Injected Applications

Now that you've seen the general implementation of DI on a component level, how do you structure an application around this concept?

Generally, apps that use DI follow an inverted hierarchical instantiation model where components with the least dependencies are built first and injected into components that need only the newly built dependencies and on and on until you have a fully configured root component. We commonly refer to this as an "object graph." This is arguably the most cumbersome part of DI.

An example of doing this manually looks like this:

// lots of imports...

export default function getDeps(config: Config): Deps {

    const db = knex(config.db)

    const saveUserPreferences = saveUserPreferencesFactory(db)
    const getUserPreferences = getUserPreferencesFactory(db)

    // ...

    const userService = new UserService(
        config.max,
        saveUserPreferences,
        getUserPreferences
    )

   // ...

   const createUserRoute = createUserRoute(userService);

    // ...

   return {
     createUserRoute,
     // ... Other top-level components used by a web server or something.
   }
}

This approach to DI is valid but very manual (and frankly annoying). The developer is responsible for resolving the correct order of dependency creation. More importantly, all components are instantiated immediately. Imagine the code base is an API server, but you wanted to write a couple of administrative command-line utilities using services/helper functions in the codebase. You probably don't want to start database connections if you don't need to. You certainly don't want an instance of the webserver starting. So how do we accomplish this when the DI pattern has us creating our object graph on initialization?

The answer is that we want lazy loading. This means a component is only created when it is needed. If the requested component depends on other components, those components are also constructed. Using our previous example, if we wanted an instance of UserService, we would also need saveUserPreferences and getUserPreferences, both of which need db.

So how do we write the lazy loading code?

Dependency Injection Frameworks

The short answer is, we don't!

Lazy loading dependencies and resolving an object graph is a non-trivial programming task. It would be silly to try to write that functionality every time we want to build an application. Instead, we can leverage various frameworks built by our community of developers that will do this for us.

In the JavaScript/TypeScript world, there are various Dependency Injection frameworks. My personal favorite is a library called Awilix.

Let's rewrite our getDeps function using Awilix.

import { NameAndRegistrationPair, asFunction, asClass, asValue } from 'awilix'

export default function getDeps(config: Config): NameAndRegistrationPair<Deps> {

    return {
        // "asValue" means "this is a constant".
        config: asValue(config),

        // "cradle" is a magic proxy that will auto resolve your dependency
        // by the "field" name in this NameAndRegistrationPair
        db: asFunction((cradle) => knex(cradle.config.db)),

        // You can even deconstruct the cradle object.
        saveUserPreferences: asFunction(({ db }) => saveUserPreferencesFactory(db)),

        // And if you use the pattern of having your factory function accept
        // an object of dependencies as it's first parameter, you can simply:
        getUserPreferences: asFunction(getUserPreferencesFactory),

        // Injection into the constructor of an object.
        userService: asClass(UserService),

        // There's actually a better way to use Awilix in libraries like
        // Express.  We will look at those in another post.
        createUserRoute: asFunction(createUserRouteFactory),
    }
}

Now when you want to resolve a component, you pull it out of the Dependency Injection context:

import { createContainer } from 'awilix'
import getDeps from './getDeps'

// assuming you have constructed the config
const deps = getDeps(config)

const container = createContainer().register(deps)

// To resolve a dependency, you can do it directly:
const userService = container.resolve('userService')

// Alternatively, Awilix makes that cool dependency proxy available
// to callers:
const { saveUserPreferences, getUserPreferences } = container.cradle

Conclusion

Using Dependency Injection will make your applications better. The pattern applies to applications of all sizes. Smaller applications probably don't need features like lazy-loading and dynamic dependency resolution and can avoid the complexity of adding a DI framework. Larger applications tend to have large and complex dependency hierarchies. Leveraging a decent DI framework can simplify the structure and maintenance of the application.

In the next post, I will demonstrate some of the magical things you can do with a decent DI framework. I hope this will encourage you to think about how you can write better apps.

Parting Thoughts

Despite DI as an essential pattern for code bases of any size, some developers gripe about the practice. If you are new to programming and unsure, I am here to dispel any doubts. I cannot think of a single, respectable software Thought Leader that thinks the DI "pattern" is bad. When engineers complain about DI, they really mean the DI "framework" they are using.

Some people follow bad practices of directly instantiating components because they think they are practicing "KISS" (Keep It Simple Stupid). These are probably the same people you inherited that shitty legacy codebase from that is impossible to maintain and as fragile as a snowflake in June.

To make the task more challenging, I decided to write the algorithm in Rust (a language I am learning but not competent in).

There are various strategies a developer might take in implementing an algorithm to determine the state of a tic-tac-toe board. Most approaches revolve around matching winning combinations (to which there are only 8) to player positions (X's and O's). The trick is determining the best way to evaluate the board state.

In this post, I'm going to walk through my thought process and implementations for two similar strategies for implementing this algorithm. You can skip all the prose and visit the actual code if you want: https://github.com/rclayton-the-terrible/solve-tic-tac-toe-rust

Model and Setup

Let's first start by establishing a simple domain model for a tic-tac-toe game:

// Represents a value at a specific position on the
// game board.
pub enum PlaceValue {
    X,
    O,
    E // Empty
}

// Represents the game board
pub struct Board {
    row1: [PlaceValue;3],
    row2: [PlaceValue;3],
    row3: [PlaceValue;3],
}

impl Board {
    // A simple method for building
    // a game board from an array of place values.
    pub fn from(positions: [PlaceValue;9]) -> Board {
        Board {
            row1: [positions[0], positions[1], positions[2]],
            row2: [positions[3], positions[4], positions[5]],
            row3: [positions[6], positions[7], positions[8]],
        }
    }
}

Our implementations (just functions) will have the following signature:

fn (board: &Board) -> Option<PlaceValue>

I have a battery of unit tests and benchmarks that will evaluate the validity of each implementation, so if you want to try your own, you will register it in src/tests.rs:

#[cfg(test)]
mod tests {
    use crate::*;
    use crate::PlaceValue::*;
    use super::test::Bencher;

    fn run_compliance_tests(eval_winner: fn (board: &Board) -> Option<PlaceValue>) {
        //... Test impl
    }

    #[test]
    fn test_bit_wise_eval_winner() {
        run_compliance_tests(bit_strategy::eval_winner);
    }

    #[bench]
    fn bench_bit_wise_eval_winner(b: &mut Bencher) {
        b.iter(|| run_compliance_tests(bit_strategy::eval_winner));
    }

    #[test]
    fn test_loop_eval_winner() {
        run_compliance_tests(loop_strategy::eval_winner);
    }

    #[bench]
    fn bench_loop_eval_winner(b: &mut Bencher) {
        b.iter(|| run_compliance_tests(loop_strategy::eval_winner));
    }
}

Loop Strategy

My first thought was to take the most direct approach, a nested loop strategy, that iterated over winning combinations and checked if the gameboard had any of those combinations for either player.

In (my sort of) pseudo-code, it would look like this:

for each winning combo [C]:
  for each value in the combo [c]:
    if current value at position is X:
      increment X matches
    if current value at position is O:
      increment O matches
  if X matches == 3
    return X as winner
  if O matches == 3
    return O as winner
default: return no winner

In the loop strategy, we define a winning combo as an array of "1"s and "0"s, where "1"s represent a required place value for that combo. I defined all winning combos like this:

const WINNING_POSITIONS: [[u8;9];8] = [
    [1, 1, 1, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 1, 1, 1, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 1, 1, 1],
    [1, 0, 0, 1, 0, 0, 1, 0, 0],
    [0, 1, 0, 0, 1, 0, 0, 1, 0],
    [0, 0, 1, 0, 0, 1, 0, 0, 1],
    [1, 0, 0, 0, 1, 0, 0, 0, 1],
    [0, 0, 1, 0, 1, 0, 1, 0, 0]
];

The implementation matches pretty closely to the pseudo code:

pub fn eval_winner(board: &Board) -> Option<PlaceValue> {
    for combo in &WINNING_POSITIONS {
        let mut x_matches = 0;
        let mut o_matches = 0;
        for i in 0..combo.len() {
            if combo[i] == 0 {
                continue;
            }
            // value_on_board just gets the value at position
            // "i", which is just the left-to-right, top-down
            // places on the 3x3 grid.
            match value_on_board(&board, i) {
                X => x_matches += 1,
                O => o_matches += 1,
                _ => ()
            }
        }
        if x_matches == 3 {
            return Some(X)
        }
        if o_matches == 3 {
            return Some(O)
        }
    }
    None
}

The loop strategy is not bad. It solves the problem directly and is not costly from a performance perspective, given that the algorithm will only experience 72 loop iterations (8 combos x 9 positions) at most.

However, with some additional complexity, we can optimize this algorithm a little bit.

Bitwise Strategy

Instead of using arrays and nested loops, we could model both winning combos and player selections as bit arrays (actually, unsigned integers).

The winning combos are now modelled in binary:

const WINNING_POSITIONS: [u16;8] = [
    0b_111_000_000,
    0b_000_111_000,
    0b_000_000_111,
    0b_100_100_100,
    0b_010_010_010,
    0b_001_001_001,
    0b_100_010_001,
    0b_001_010_100
];

The algorithm changes slightly. We don't need to loop through every position on the board multiple times to determine the winner:

convert board positions for X's and O's into bit arrays
for each winning combo (as int):
   if (X & combo) == combo:
     return X as winner
   if (O & combo) == combo:
     return O as winner
default: return no winner

The key to this approach is the bitwise & (AND) operator.

The bitwise AND operator is applied to two numbers (e.g. 402 & 163). All 1 bits shared at the same position between both numbers are retained. All other positions are filled with 0s:

We already have all the winning combo positions defined as integers. All we need to do is convert the board positions for X's and O's into integers as well:

struct XPosOPos(u16, u16);

fn board_to_bits(board: &Board) -> XPosOPos {
    let mut x_bits: u16 = 0;
    let mut o_bits: u16 = 0;
    for i in 0..9 {
        let row = get_row(&board, i);
        let position = get_row_pos(i);
        match row[position] {
            PlaceValue::X => x_bits |= 1 << i,
            PlaceValue::O => o_bits |= 1 << i,
            _ => (),
        }
    }
    XPosOPos(x_bits, o_bits)
}

The functions get_row and get_row_pos are just helpers that translate i (the index in our 9-bit array) to the 3x3 grid of tic-tac-toe. With that position, we determine whether the value is "X", "O", or "E" (empty). If it's X or O, we need to set the bit at position i in that integer to 1. We can set the bit using the bitwise | (OR) and the left shift << operators.

The bitwise OR operator is applied to two numbers (e.g. 402 & 163). For all bit positions, if a 1 occurs in either number, the value of that bit position becomes 1. All positions where both numbers have a 0 remain 0:

The left shift operator << is an operator that shifts all bits on the operator's left-hand side by N positions (specified by the value on the right-side). For example, 0b001 << 1 results in 0b010 or 2.

When the bitwise OR (|) and left-shift (<<) operators are combined, you can set the bit at a specific position in a number. To do this, we make a "mask" -- a bit array with the 1 in the correct position. We then perform a bitwise OR between the number whose bit we want to set and the mask. For example, say we want to set the 6th bit to 1:

let mut n = 22;
// move the 1 bit to the 6th position; 0b000_000_001 becomes 0b000_100_000
let mask = 1 << 6;
n = n | mask;
// Alternatively, Rust has |= operator similar to +=
n |= mask;

Now that we understand how the board state is translated into two integers (bit arrays) representing the X and O positions, let's look at the algorithm for determining the winner:

pub fn eval_winner(board: &Board) -> Option<PlaceValue> {
    let XPosOPos(x_bits, o_bits) = board_to_bits(&board);
    for pos in &WINNING_POSITIONS {
        if (pos & x_bits) == *pos {
            return Some(PlaceValue::X);
        }
        if (pos & o_bits) == *pos {
            return Some(PlaceValue::O);
        }
    }
    None
}

All we do now is iterate over combo positions and check to see if the combo matches the X and O positions ((pos & x_bits) == *pos).

It's easier to see what's happening with this expression by looking at an example:

(0b010 & 0b111) == 0b010
// simplifies to:
0b010 == 0b010

Comparison of Strategies

The loop and bitwise strategies are both effective. As I mentioned before, the worst-case scenario for the loop strategy is 72 loop iterations. The bitwise method accomplishes the same result with a maximum of 17 loop iterations (9 to create the X and O bit positions and 8 to loop through the combos). Of course, the body of the loops are also different - so perhaps not an apt comparison.

In terms of memory, the bitwise approach uses fewer resources. The loop approach has to store 72 integers (8, 9 element arrays) plus two integers for the intermediate state during iterations. The bitwise strategy uses eight integers for combo state and two integers for the X and O board state.

Both approaches use a minimal amount of memory and would be acceptable in a production system. The loop method is arguably easier to maintain. However, if you are interviewing or completing a Leetcode challenge, you will want to optimize for the fewest resources.

So what do the benchmarks look like?:

> cargo bench
    Finished bench [optimized] target(s) in 0.00s
     Running unittests (target/release/deps/solve_tic_tac_toe-2ba3cdb4b32fcc5e)

running 5 tests
test bit_strategy::tests::test_board_to_bits ... ignored
test tests::tests::test_bit_wise_eval_winner ... ignored
test tests::tests::test_loop_eval_winner ... ignored
test tests::tests::bench_bit_wise_eval_winner ... bench:          97 ns/iter (+/- 4)
test tests::tests::bench_loop_eval_winner     ... bench:         581 ns/iter (+/- 44)

test result: ok. 0 passed; 0 failed; 3 ignored; 2 measured; 0 filtered out; finished in 10.38s

The bitwise approach is nearly five times faster!

Conclusion

I thought this was an exciting project to apply Rust and optimization techniques like bitwise operators. Tic-tac-toe is not the most practical thing to optimize for, but the solutions demonstrate bitwise operators' potential if you need to eke out extra performance from some code.

If you thought the bitwise operators were neat, please check out this course on Educative: Master Solving Problems using Bit Manipulation.

Despite the increasing awareness of observability practices, there remain many barriers to adopting these technologies. Perhaps the most significant obstacle is integrating libraries/frameworks into services. For instance, it's unlikely every observability tool will have a client for your platform.

Relying directly on clients can also be a problem. Observability tools differentiate themselves from competitors by offering unique features. If you use those features, it will be hard to move off the client if your new provider does not have a similar feature set. Instead, your service could rely on a standardized abstraction. For example, OpenTelemetry offers client abstractions for Tracing and Metrics collection. Some languages, like Java, have similar abstractions for logging (SLF4J).

The biggest issue with client integrations is how intrusive the implementation can be in your domain code. Look at the example from the OpenTracing website:

// https://opentelemetry.io/docs/js/instrumentation/

function doWork(parent) {
  const span = tracer.startSpan('doWork', {
    parent, attributes: { attribute1 : 'value1' }
  });
  for (let i = 0; i <= Math.floor(Math.random() * 40000000); i += 1) {
    // empty
  }
  span.setAttribute('attribute2', 'value2');
  span.end();
}

Of course, there are ways to abstract some of these "crosscutting concerns," but that assumes the framework in which you write code is flexible enough to be modified to perform some of this work. For instance, we implemented an "auto span" plugin for Knex.js to transparently record database calls in OpenTracing (Jaeger backend). However, implementing the plugin required a service library with an explicit extension point (something you will not get out of the box with many HTTP frameworks).

So far, I have spoken mainly about the "client-side" integration pattern for observability tools, which looks something like this:

Most architectures don't use pure client integration anymore. Aggregating data from the client directly to the backing store doesn't make sense with thousands of processes in your infrastructure. Client-side aggregation suffers typical operational challenges:

• Configuration management (including refreshing for running services)
• Service discovery (where is the observability endpoint?)
• Reliability (throttling, circuit breaking, retries, etc.)
• Security (should the service be communicating to the endpoint?)

Client-side integration also has the downside of placing a lot of the logic (and therefore processing burden) on the local process. So not only is the service coordinating actions for its clients, but it's also managing connections to Statsd, serving Prometheus metrics requests, formatting logs, etc.

Standardizing on Daemons/Sidecars

The general solution to client-side integration problems is to move the processing to a more centralized service. The industry has developed two standard deployment patterns to support this use case: daemons and sidecars:

A daemon is an external service that provides functionality to a traditional business application. In this case, the daemon (Fluentd) parsing and aggregating logs. Daemons deploy as single-instance services on a host (bare-metal or VM). Fluentd, a log aggregator, will consume all the stdout and stderr streams in Docker and forward the entries to one or more external endpoints.

Sidecars deploy with an application (think "sidecar" on a motorcycle) and only serve that peer application. A typical use case for sidecars is to provide secrets and configurations specific to its peer service.

The daemon and sidecar patterns are more effective when providing multiple services (e.g., logging, metrics, tracing). However, until recently, few open-source frameworks offered multiple observability modes. The first multi-service daemons were commercial offerings like Datadog:

Datadog's integration, however, is far from perfect. Services are still required to use multiple strategies/protocols to aggregate data to the daemon. For instance, Datadog will collect logs from stdout but still requires services to use the Statsd protocol for metrics and the OpenTelemetry library for traces. What makes the integration more painful is that none of these tools integrate out-of-the-box. Developers have to propagate IDs between logs and traces to correlate them in the dashboard.

I generally think the daemon pattern (and not the sidecar) is the best pattern for providing observability into services. I also think having the daemon own the entire process is the ideal approach for integrating services. The only thing missing is a unified mechanism for delivering logs, metrics, and traces to the daemon.

The key to the unified integration pattern is that the service, for the most part, is unaware of it. Instead, services publish a standardized event to an "observability stream." The event can be a log, trace, or metric. More likely, the event is all three; it blurs the distinction which better models reality and not the separation of tools.

Aggregation is a result of a daemon monitoring the observability stream. The daemon would interpret events, transforming them if needed, and forward them to the correct backing store. More importantly, if backing stores can perform cross-tool correlation, the daemon has the context necessary to provide the correlating identifiers.

If the standard way of producing observability events is stdout (the console), virtually any system could integrate with the aggregation system. Using stdout could be a problem in systems with large capacity hosts and a high density of services. An alternative approach might be using Kafka as the event stream.

Either way, a stream of observability events has the advantage of allowing multiple providers to listen to a stream. I would discourage having a daemon for each observability mode (logging, metrics, tracing). However, using different daemons as means of testing or transitioning between providers would be a powerful paradigm. Imagine being able to launch an AWS daemon alongside your Datadog agent so you could explore the newest features of CloudWatch or XRay. If you were using a persistent stream, you could even migrate old data into the new system with little effort.

Standard Observability Event

Our ability to get to a unified observability stream would require the industry to standardize on an event schema. I think now, more than ever, we may have the impetus in the industry to reach this goal. OpenTelemetry and CloudEvents are great examples of organizations coming together under a standard. CloudEvents would probably be the envelope schema used by a Unified Observability Event.

So what would the developer experience be like with unified observability events?

The best solution would look like logging:

log.info('Request received to update user profile.')

Admittedly, this is not a very impressive example. However, when combined with context from the request:

export default [
  // Of course, this should be registered at the app level,
  // unless there are request-specific metadata to add.
  observe.startContext({
    component: 'UpdateUserProfile',
    tags: {
      add: {
        foo: 'bar',
      },
      fromEnv: ['CLUSTER', 'NODE_ENV'],
    },
  }),
  (req, res, next) => {
    req.log.info('Request received to update user profile.')
    // ...
  }
]

The log.info would emit to stdout an event that looked something like this:

{
    "specversion" : "1.0",
    "type" : "org.cncf.observable/msg",
    "source" : "com.myco/users",
    "subject": null,
    "id" : "34dc3d61-6c71-4694-9c59-f354d327dce7",
    "time" : "2021-03-21T04:02:21.331Z",
    "datacontenttype" : "application/json",
    "data" : {
        "id": "34dc3d61-6c71-4694-9c59-f354d327dce7",
        "time" : "2021-03-21T04:02:21.331Z",
        "level": 30,
        "component": "UpdateUserProfile",
        "message": "Request received to update user profile.",
        "tags": {
          "foo": "bar",
          "CLUSTER": "staging",
          "NODE_ENV": "staging"
        },
        "metadata": {
          "sessionId": "121jkhuaiuuhaiusdh13989hasjkdfh1",
          "params": {
            "userId": 123,
          },
          "client": {
            "agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X...",
            "ip": "127.0.0.1"
          }
        }
    }
}

This is pretty-printed -- it would be one object per line.

Observability contexts originate using the common "child" convention. Child contexts inherit metadata from the parent and can override/extend those properties with additional tags. Child contexts are also a natural place to begin and end tracing spans automatically:

async function getUserProfile(
  logger: Log,
  userId: number,
): Promise<Profile | null> {

   const log = logger.child({
     component: 'getUserProfile',
     userId,
   })

   log.debug('Looking up user profile.')

   const profile = await repo.getProfile(userId)

   log.debug(
     { profileFound: !!profile },
     'Returning profile to caller.'
   )

   // It would be nicer to auto close this!
   log.finish()

   return profile
}

// ...
(req, res, next) => {
   // ...
   const profile = await getUserProfile(
     req.log,
     req.params.userId,
   )
   // ...
}

The code above might produce the following events:

[
  {
      "specversion" : "1.0",
      "type" : "org.cncf.observable/msg",
      "source" : "com.myco/users",
      "subject": null,
      "id" : "34dc3d61-6c71-4694-9c59-f354d327dce7:d1f358c60c4e:1",
      "time" : "2021-03-21T04:02:21.012Z",
      "datacontenttype" : "application/json",
      "data" : {
          "level": 20,
          "component": "getUserProfile",
          "message": "Looking up user profile.",
          // ...metadata and tags
       }
  },
  {
      "specversion" : "1.0",
      "type" : "org.cncf.observable/msg",
      "source" : "com.myco/users",
      "subject": null,
      "id" : "34dc3d61-6c71-4694-9c59-f354d327dce7",
      "time" : "2021-03-21T04:02:21.212Z",
      "datacontenttype" : "application/json",
      "data" : {
          "level": 20,
          "component": "getUserProfile",
          "message": "Returning profile to caller."
          // ...metadata and tags
      }
  },
  {
      "specversion" : "1.0",
      "type" : "org.cncf.observable/ctx-end",
      "source" : "com.myco/users",
      "subject": null,
      "id" : "34dc3d61-6c71-4694-9c59-f354d327dce7:d1f358c60c4e:3",
      "time" : "2021-03-21T04:02:21.323Z",
      "datacontenttype" : "application/json",
      "data" : {
          // ...metadata and tags
      }
  }
]

The example events use a hierarchical ID to represent the context chain. The original ID (a UUID) originated from the request middleware. The last event's message type signifies the end of a context; this would be used by tracing systems to mark the end of a span.

Conclusion

While observability tools (logging, metrics, tracing) continue to evolve, the developer experience and infrastructure continue to lag behind vendor solutions. I believe the industry could make some minor changes towards a unified standard for emitting "observability events" that would be easier (and cleaner) to integrate into services and decouple aggregation infrastructure and vendor solutions. This approach would benefit all parties by minimizing the complexity of using these tools and democratizing the interfaces so vendors and open-source providers could compete on a level playing field.