It is funny enough that I feel like I have to say that I hate writing and I am so happy that I finally sitting down and typing out these first words.

Back in 2025 I had a big burnout and several good things came out from it. One of them is that in the late autumn I transitioned from a full-time developer behind a legacy application to pioneering the AI integrations and automations at the juice company.

Many laugh thinking that I joke when I say I do that because "why would a food company need AI" which is quite funny because it might the type of a company that can benefit the most.

I have been building various workflows with n8n since October and it has taught me a lot. I have done several things with AI so it was not a complete new thing but there were many new learnings and I am excited to share them with you in my diary.

Steep learning curve with n8n

n8n is a more technical automation/integration platform, similar to make.com or Zapier however allowing to dive deeper with the code blocks and data mutation outside the predefined generic nodes.

I am as much of a developer as one can be. I seriously got into it when I was 16 years old when I was determined to use my summer vacation to build a web app - I did it. I love writing code because it's a first language to me and it feels so natural. That is not how I felt with n8n.

It required my brain to sweat quite a bit. There are many subtle pitfalls that only reveal themselves once you start building real workflows. There are probably countless of videos on the YouTube and articles out there teaching about those and I may have happened to skim through those parts. Maybe not. I will share some of those learnings here on my blog.

The biggest change is that in the code, let's say JavaScript since everyone is familiar, it is super easy to control concurrency. One either awaits one function or promise, or entire group of functions/promises. One explicitly defines where the data flows and what is returned. One can optionally enrich the data or flow via one or the other function without cracking the head. This is not so easy in n8n. Here are a few examples.

Few pain points with n8n workflows

Return data

Vertical order in n8n does matter, if there are two final nodes in the sub workflow and the one you don’t care is below the one you do care, it will output the wrong data. I would love that for enterprise approach one could explicitly select which return node is the actual return node and not depending on the order hierarchy.

The entry point is easy, the return is easy if there is only one node. However a good practice is to use the Execution Data node which lets you annotate your execution with various parameters - amazing for debugging. Now you have two nodes that are end nodes. The return data from the sub-workflow execution will be from the one that is the bottom-most. There is no explicit way to tell that in n8n. This makes me question whether n8n is truly enterprise‑ready in its current form. Big gotcha.

Nodes in n8n execute from left to right and from top to bottom. There also isn't any real concurrency if the parent execution wants to keep the track of. Meaning it's always left top to right bottom.

This bit me quite a lot. I feel itchy like after feeding an army of mosquitos in nature. I love converting bigger chunks of the workflow to a sub-workflow as soon as it makes sense to reuse or a soon as it logically means it can be encapsulated.

Be generous with logging

Debugging can be very cumbersome. I wish I could plug the errors to Sentry via some native connector but unfortunately that doesn't exist out-of-the-box.

Using the aforementioned Execution Data node is a must. In this node I always log the primary ID of the item I am processing. Such as Zendesk Ticket ID for the tickets there or the internetMessageId for the emails in Outlook.

Besides that I log extra information that I believe would make sense. Such as if your workflow has several branches (think processing extra images if exist) and whether this workflow has run that branch. You can have as many nodes along the workflow as you wish to enrich it.

One self decides on that. However searching for executions and filtering by the id is a game changer.

Do not use Cmd + Z (or Ctrl + Z) // Undo more than once

This is why I love computers. This is such a treat that one can try, go back and redo. People in sales, in medicine, in buildings can't quite do that. Fuck around and find out is how I got in my career where I am.

However NEVER click too many times Cmd - Z after dragging things around. My entire workflow got broken several times. I ended up experiencing vague execution errors saying the node doesn't exist or the data is wrong while iterating in the development. Downloading and re-importing the workflow or copying the nodes to another didn't help.

I had to go almost node by node in a new workflow and rebuild it from scratch. Now I fear the Undo like touching the hot pot.

No first party nodes for tracking the AI usage, tokens, or raw responses

This is huge. If not AI, n8n would be a small kid playing outside with anyone barely knowing its name. AI enabled people to build mini automations augmented with LLMs that bring enough value for it to blow up.

Crafting workflows in the enterprise, ideating, iterating and prototyping, for someone with ML training background, is a no-brainer to log the input/output for the AI that later can be used for Evals, for fine-tuning, for error or cost insights.

To this day, using the latest version at the day of writing, there is no node or clean way to store ai exchanges.

My first solution was to call the LLM node such as Claude to ClaudeAiClient and have sub workflow execute after the parent finishes (with no await for completion). The sub workflow would then retrieve the execution, find the *AiClient node and extract the messages and store to the database. This way I had exact tokens, raw messages and everything one would need.

Eventually our workflows have evolved a lot and due to other things I won't discuss here, I gave up on that detail logging and now just store the clean output in the main workflow.

I dream that n8n would add a final node that allows to retrieve all ai exchanges for sub nodes and let us handle the data how we wish.

Enterprise loves to answer the question how much does classifying the tickets cost? Okay, does it bring enough value to pay for them? How about the writing the response? How much does that cost? Oh that is 10M of tokens a day.... You get the gist.

The pleasure points with n8n

I had to get my complaining out of the way so I can focus on the pleasure. The n8n is an amazing platform for prototyping once the steep curve is ironed out.

Quick iteration

A lot of these workflows are novel. AI is still novel. Less than 3 years ago we heard about GPT 4 coming out which was slow and stupid. Now I trust ChatGPT to do my SKAT book keeping because it's that good at deep research and comprehension.

Same applies in the corporate. We know that we want the help at X department by doing Y. But the stakeholders don't necessarily know all that it takes and most importantly all that it takes they do implicitly. It is such an important thing that applies to us all.

If I say I am gonna eat an apple, I will implicitly wash the apple and wash my hands. I will also take the knife and cut the apple to slices because I wear braces. Think about this. There are many more steps and conditional processing. If one has braces on - the apple needs to be cut. If the email has big and many attachments, they need to be pre-processed before the AI can read the email and take a decision.

The hard thing here is that often some of these steps are only discovered once the thing is running. Walking up with the laptop to review the workflow with a non‑technical stakeholder and looking at the outputs at various stages - is like fine‑tuning a radio. It becomes visual, intuitive, and collaborative. That's a big plus.

Various Trigger nodes, aka integrations with platforms

Even though I still want to write code for some of these, having the built in triggers such as Zendesk for new incoming tickets, it's a bliss. I don't need to deploy the code to the cloud, I don't need to go in the Settings and register the webhook, I don't need configure credentials (besides putting ones in the n8n for the node). It just works, most of the time.

The gotcha here is that one has to be careful of the events. Too many executions and one can blow away through the monthly allowance. So be careful. Otherwise, it is a bliss to iterate with that.

I have done some funny stuff with my own Strava, an app for storing the runs or bike rides.

The greatest benefit of building these automations

Documentation.

Despite the common questions such as are we now trying to replace people with AI. In my case and journey - absolutely not.

However what I really love about building these workflows is how much implicit documentation comes out. I am now working with three prototypes for the support center and handling a shared mailbox. Since the first run of the automation, I've spared a lot with my colleagues to improve and explain AI (aka: write the system prompts) what, why and how should XYZ be done. Ever since, AI has caught up by catching human error where the people at question don't follow their own rules or things they explained. That means two things - they didn't tell everything at first because for us humans, things become a reflex muscle that we perform subconsciously and others, we actually just forget and do a mistake based on the moon phase.

Thanks to these workflows, I have some of the best documentation for certain touch points than any other departments because if this is not the "best", it won't perform its duty and deliver the value.

AI cannot be trusted

This is the final point that I feel is very important to note down. The enormous amount of hype being pumped on a regular basis by the major LLM developers (OpenAI, Anthropic, Google) as well as the entire AI Agency community and SaaS builders is insane. My former manager and a friend has caught me stuck in that bubble back in 2023. Now around three later, AI is a helpful tool but it fails in ridiculous ways.

Once every a few hundred executions given the task where it must output structured data with two fields - a string reasoning field and an integer correct_output field being - a union of either 1 or 2 - it will output 0 or 3. This is easy to show, but the funny/sad part is where the reasoning above is correct and the model still produced the wrong choice despite its own very reasoning.

Model in question has mostly been Claude Sonnet 4.5. The temperature by default is 0 aiming for the most consistent predictable outputs. The tricky part is that rerunning the same input will not cause the same error thus rendering any debugging basically useless. If it failed with illegal data? Simply rerun.

I have a few more examples that I will share in the future articles but this I find to be so small and so trivial task that it should never fail, in my perception.

Final remarks

I envision writing at least two more posts, based on how in-depth I am allowed to dive with the things I am engineering at the sandwich and juice company. One of them is about handling a shared Outlook Inbox that receives as many 5000+ emails a week, of which majority need to be accurately classified and stored, and a minority requires the human in the loop. The other being, classification and assistive reply support for the Customer Care on Zendesk system. Both are very interesting cases and have some exciting learning. Both have been running for several months and both departments are accustomed to have it running now.

If you're a human and read this far, I'd love to hear from you in the comments - are you exploring to use n8n? Are you evaluating the entire automation industry? Are you just learning? Or are you a seasoned hacker/tinkerer like me? :)

Background

Basically one of my side projects is this extension for the flame looking swiping app - Hard to guess, I know! The APIs provided for it is hosted on the Cloudflare Workers. Amazing service for the value it brings just for a few dollars a month.

Since the early days, once I discovered Vercel’s AI SDK I started using it as it seems like an absolute no brainer to offer beautiful streaming experiences while keeping majority of flexibility of choosing models and providers and ofc designing the UI.

When version 5 came out and I looked into it first time end of August, it felt completely overwhelming. I was recovering from burnout at work and my brain wasn’t fully functional. I tried running the codemod it failed and so I left it.

Fast forward to the beginning of October, I am doing so much better, and looked into it again and actually it seemed really quite simple this time.

My biggest breaking changes

To preface, I created a new git worktree for both backend and frontend where I started to integrate things step by step focusing on as few changes as possible.

The v5 response uses Server Sent Events (SSE) & pipeV5StreamToV4Response

If you are new to SSE, check out MDN docs at https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events

This is by far the biggest breaking change and everyone would be affected. All versions up to the v5 were using Vercel’s proprietary invention, which was either basic text-only stream or a more complex data stream supporting tool calling etc. It’s actually pretty interesting and cool thing. If you’ve never written one yourself, feel free to check out their docs. The image below is from their v4 docs.

v4: https://v4.ai-sdk.dev/docs/ai-sdk-ui/stream-protocol#stream-protocols

v5: https://ai-sdk.dev/docs/ai-sdk-ui/stream-protocol#stream-protocols

Frankly, as cool as it is, I have no idea why they chose it in the first place. SSE was already available at the ChatGPT release in 2022 and I have even written a stream in PHP backend. Although this is exactly what I’m guilty of, reinventing the wheel. It seems cool and exciting at first but then it brings the maintenance burden and limitations. For what’s worth it - I really do appreciate SSE protocol.

I was looking into how I could rewrite the responses. The text protocol was out the window since I already use tool calling. Fortunately no images at this point via messages.

Lucky me, and I really mean it, someone shared a snippet of code that remaps the v5 message parts to v4 on project’s GitHub issue. The function is called pipeV5StreamToV4Response Once I tested that it actually works, I only needed to detect the client version. I went ahead and updated client headers to indicate whether it’s on a new version or old one.

This is how I plugged it into my backend API

// in /src/functions/ai/useAI.ts

...
// clientVersionValue comes from headers, provided from arguments 
const convertToV4 = clientVersionValue !== 'v5';

 const result = streamText({
    temperature: 0.6,
    maxOutputTokens: 1_000,
    messages: modelMessages, // provided from arguments
});


if (convertToV4) {
    const stream = result.toUIMessageStream({ onError });
    return pipeV5StreamToV4Response(stream, {
        headers,
    });
}

return result.toUIMessageStreamResponse({
    onError,
    headers,
});

// in /src/functions/helpers/pipeV5StreamToV4Response.ts
import type { UIMessageChunk } from 'ai';

// adapted from
// https://github.com/vercel/ai/issues/7993#issuecomment-3180974654
// credit for Mastra.AI for writing this piece of wonderful code
// also related:
// https://github.com/davidondrej/AI-CRM/blob/e7e3848a6ec99f4eee2c29d9208bd4bf11b0f6c9/crm-app/docs/v5-vercel-sdk.md

/** Pipes an AI SDK v5 UIMessage stream to a response in a v4 compatible format. */
export function pipeV5StreamToV4Response(
    stream: ReadableStream<UIMessageChunk>,
    responseInit?: ResponseInit,
): Response {
    const v4Response = createV4Response(stream, responseInit);
    return v4Response;
}

function createV4Response(
    v5Stream: ReadableStream<UIMessageChunk>,
    responseInit?: ResponseInit,
) {
    const v4Stream = v5Stream
        .pipeThrough(createV5ToV4Transformer())
        .pipeThrough(new TextEncoderStream());
    return new Response(v4Stream, {
        status: 200,
        headers: {
            'Cache-Control': 'no-cache',
        },
        ...responseInit,
    });
}

type StreamState = {
    messageCounter: number;
    stepCounter: number;
};

function createV5ToV4Transformer() {
    const state: StreamState = {
        messageCounter: 0,
        stepCounter: 0,
    };
    return new TransformStream<UIMessageChunk, string>({
        transform(chunk, controller) {
            try {
                const v4Chunk = transformV5ChunkToV4(chunk, state);
                if (v4Chunk) {
                    controller.enqueue(v4Chunk + '\n');
                }
            } catch (transformError) {
                // noop
            }
        },
    });
}

/** Map of v5 stream prefixes to v4 stream prefixes. */
const DataStreamStringPrefixes = {
    text: '0',
    data: '2',
    error: '3',
    message_annotations: '8',
    tool_call: '9',
    tool_result: 'a',
    tool_call_streaming_start: 'b',
    tool_call_delta: 'c',
    finish_message: 'd',
    finish_step: 'e',
    start_step: 'f',
    reasoning: 'g',
    source: 'h',
    redacted_reasoning: 'i',
    reasoning_signature: 'j',
    file: 'k',
};

function transformV5ChunkToV4(chunk: UIMessageChunk, state: StreamState) {
    switch (chunk.type) {
        case 'text-start':
            return null;
        case 'text-delta':
            return `${DataStreamStringPrefixes.text}:${JSON.stringify(
                chunk.delta,
            )}`;
        case 'text-end':
            return null;
        case 'error':
            try {
                const errorData =
                    typeof chunk.errorText === 'string'
                        ? {
                              message: chunk.errorText,
                              code: 'STREAM_ERROR',
                              timestamp: Date.now(),
                          }
                        : chunk.errorText;
                return `${DataStreamStringPrefixes.error}:${JSON.stringify(
                    errorData,
                )}`;
            } catch {
                return `${DataStreamStringPrefixes.error}:${JSON.stringify({
                    message: 'Stream error',
                    code: 'TRANSFORM_ERROR',
                })}`;
            }
        case 'tool-input-start':
        case 'tool-input-delta':
            // Don't stream partial tool calls for v4
            return null;
        case 'tool-input-available':
            return `${DataStreamStringPrefixes.tool_call}:${JSON.stringify({
                toolCallId: chunk.toolCallId,
                toolName: chunk.toolName,
                args: chunk.input,
            })}`;
        case 'tool-output-available':
            return `${DataStreamStringPrefixes.tool_result}:${JSON.stringify({
                toolCallId: chunk.toolCallId,
                result: chunk.output,
            })}`;
        case 'reasoning-start':
            return null;
        case 'reasoning-delta':
            return `${DataStreamStringPrefixes.reasoning}:${JSON.stringify(
                chunk.delta,
            )}`;
        case 'reasoning-end':
            return null;
        case 'start':
            return null;
        case 'finish':
            return `${DataStreamStringPrefixes.finish_message}:${JSON.stringify(
                {
                    finishReason: 'stop',
                    metadata: chunk.messageMetadata,
                },
            )}`;
        case 'start-step':
            state.stepCounter++;
            const stepId = generateId(`step-${state.stepCounter}`);
            return `${DataStreamStringPrefixes.start_step}:${JSON.stringify({
                messageId: stepId,
            })}`;
        case 'finish-step':
            return `${DataStreamStringPrefixes.finish_step}:${JSON.stringify({
                finishReason: 'stop',
                usage: {
                    promptTokens: 0,
                    completionTokens: 0,
                },
                isContinued: false,
            })}`;
        case 'message-metadata':
            return `${
                DataStreamStringPrefixes.message_annotations
            }:${JSON.stringify([
                {
                    messageId: generateId('msg'),
                    metadata: chunk.messageMetadata,
                },
            ])}`;
        case 'file':
            if ('url' in chunk && 'mediaType' in chunk) {
                return `${DataStreamStringPrefixes.file}:${JSON.stringify({
                    url: chunk.url,
                    mediaType: chunk.mediaType,
                })}`;
            }
            return null;
        case 'source-url':
            return `${DataStreamStringPrefixes.source}:${JSON.stringify({
                id: chunk.sourceId,
                title: chunk.title,
                url: chunk.url,
            })}`;
        case 'source-document':
            return `${DataStreamStringPrefixes.source}:${JSON.stringify({
                id: chunk.sourceId,
                title: chunk.title,
                mediaType: chunk.mediaType,
                filename: chunk.filename,
            })}`;
        default:
            // Unknown chunk type, ignore
            return null;
    }
}

function generateId(prefix: string) {
    return `${prefix}-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`;
}

The biggest thing solved. I now have streaming working with both the new and old clients.

Mixed up types & Remapping messages with attachments

The following part of migration was remapping the attachments. I began using those to send the match pictures a long to the model provider so LLMs can see the profile as well.

It took me a good hour of deep focus to carefully read documentation on both versions. Vercel did change things up and there was a lot of mixing between mediaType, mimeType and just type for providing the mime type, as well as the content and it got me really confused. I don’t quite know if I missed something or the team got wrangled up in their own types.

For example, in v4 they had experimental_attachments with FileList since I provide URLs and don’t host those images myself, I need either an encoded data-uri or publicly accessible URL. So I used Attachment in v4 and FileUIPart in v5.

v4: https://v4.ai-sdk.dev/docs/ai-sdk-ui/chatbot#attachments-experimental

v5: https://ai-sdk.dev/docs/ai-sdk-ui/chatbot#attachments

Example objects between versions

// version 4 
{
      name: 'earth.png',
      contentType: 'image/png',
      url: 'https://example.com/earth.png',
 }

// version 5
 {
      type: 'file',
      filename: 'earth.png',
      mediaType: 'image/png',
      url: 'https://example.com/earth.png',
 }

One confusing aspect was the FileUIPart type, which appears in multiple packages with different signatures. Vercel - but why???

During branch merging between versions I came to find out that actually one can import FileUIPart part from different packages. Take a look, the identical type however has a completely different signature.

import type { FileUIPart } from '@ai-sdk/ui-utils';

and

import type { FileUIPart } from 'ai';

Perhaps it wasn’t a big deal doing so in v4 as the backend somehow worked (I didn’t dive deep) and was prep-ing for the v5. However upon updating the client, I had to do quite a bit of digging to find out that I was fooling myself with a wrong type all along.

Upon sharing this on twitter, Lars Grammel explained that @ai-sdk/ui-utils shall not be used for v5 (it remains active for v4). Chances are the node_modules folder wasn’t clean and hence I pulled it in accidentally.

The image below is preparing my client running v4.

and updating the options argument for the submit - sendMessage method:

As for the backend, in v5 I had to write a tiny little remapping to inject messages as message parts

// Start of v4 remapping -----------------------------------------------------
// This is only received from v4, delete once all clients are on v5
const remappedMessages = isSdkV4
    ? messages.map((message) => {
            const parts = message.parts || [];

            if (
                'experimental_attachments' in message &&
                Array.isArray(message.experimental_attachments)
            ) {
                message.experimental_attachments.forEach((attachment) => {
                    parts.push({
                        type: 'file',
                        mediaType: attachment.contentType,
                        url: attachment.url,
                    });
                });
                delete message.experimental_attachments;
            }

            return {
                ...message,
                parts,
            };
        })
    : messages;

const modelMessages = convertToModelMessages(remappedMessages);
// console.log({ modelMessages, messages });
// End of v4 remapping

const res = streamText({
    ...
    messages: modelMessages,
    ...
});

For the clarity, in v4 the experimental_attachments are automagically handled inside streamText function - it just works! In v5 I have to remap using the snippet above into the parts before applying convertToModelMessages on the messages and sending to the streamText function.

Perhaps, after reading this section, you can see how many types there are. If I had to work on this migration more, I would create a mind map with all the types and the way they are used to have a clarity at sight. If I got as many bonuses as how many keys there are to send the mimeType of the attachment/image, I’d be on vacation in Hawaii haha.

Worth a mention, the attachments had to be rendered by me in UI and they were not tied to any message by the useChat logic - it was up to me, the developer, to decide where and how I render them. In v5 they became the part of the message and would be rendered as a file type:

{
  messages.map(message => (
    <div key={message.id}>
      {message.parts.map((part, index) => {
        if (part.type === 'text') {
          return <div key={index}>{part.text}</div>;
        } else if (
          part.type === 'file' &&
          part.mediaType.startsWith('image/')
        ) {
          return <img key={index} src={part.url} />;
        }
      })}
    </div>
  ));
}

This is a perfect intro to transition to the next and the last major section.

Update frontend client to use message.parts instead of message.content

This is technically not a breaking change because this was already available in v4 and I was partially using it in the frontend - but not everywhere. So I had to go over each place and ensuring I no longer relied on the deprecated message.content property for rendering.

However, there were a change in the way the parts are made and rendered.

It became more of a free-form array with support for various parts and I strongly agree with that type of a design. It’s very SOLID.

docs: https://ai-sdk.dev/docs/migration-guides/migration-guide-5-0#content--parts-array

For example, tool calls now include tool name and reasoning is now a part matching the standard reply in itself. This makes designing re-usable components so much more intuitive and simple.

// version 4
{
  message.parts.map(part => {
    if (part.type === 'tool-invocation') {
      return <div>{part.toolInvocation.toolName}</div>;
    }
  });
}
{
  message.parts.map((part, index) => {
    if (part.type === 'reasoning') {
      return (
        <div key={index} className="reasoning-display">
          {part.reasoning}
        </div>
      );
    }
  });
}

// version 5
// Type-safe tool parts with specific names
{
  message.parts.map(part => {
    switch (part.type) {
      case 'tool-getWeatherInformation':
        return <div>Getting weather...</div>;
      case 'tool-askForConfirmation':
        return <div>Asking for confirmation...</div>;
    }
  });
}
{
  message.parts.map((part, index) => {
    if (part.type === 'reasoning') {
      return (
        <div key={index} className="reasoning-display">
          {part.text}
        </div>
      );
    }
  });
}

The easy stuff

Using codemod would import old Zod version

I have no idea why but when I tried the npx @ai-sdk/codemod v5 to update the backend, it’d update all Zod imports to import the version 3 - import * as z from "zod/v3";. It happened before and after my own migration. I had already upgraded to the version 4 in package.json. Might be a bug or some stale cache issue. Who knows. I just had to revert those lines.

Overall, codemod was only helpful to see that I haven’t missed anything myself but by means was it a replacement for entire work of mine.

Keep a sharp eye on that.

Updating small renames

The rest of changes were mostly easy such as changing maxTokens to maxOutputTokens, renaming coreMessages to modelMessages and so on so far.

Fortunately there weren’t many variables that needed to be updated so I am really grateful for that.

Those are well documented in the official migration guide: https://ai-sdk.dev/docs/migration-guides/migration-guide-5-0.

I think my biggest score is that I never stored these messages in the database. The chats are sort of ephemeral. They’re cached on the client and in the transit to an extent but that’s it. I really feel like I scored a ton for not needing to migrate database schemas or alike.

Summary

It was somewhat overwhelming at first but after a weekend and 10-20 hours of deep focus, I was done and very happy.

If I were to do it again, I don’t think I’d change anything major. My biggest lesson was to keep changes in previous versions as up-to date as possible (i.e. adopt message parts instead of dot-content or dot-reasoning).

My biggest payout in my project repositories came out - keep complexity simple and have some abstraction between someone else code (the AI SDK - in this case) and my own code logic. That is, while my extension supports multiple platforms such as whatsapp, tinder, bumble, I have shared set of methods that accept a standardized structure by the provider and thus I only need to handle this set of methods to adapt to the new structure to the AI SDK. It’s like a mini proxy.

Typically, I feel lazy to write those on small projects but as soon as there are multiple sources, it is such a no-brainer investment. If one day I wanna ditch Vercel to use any other proxy, it’d not be too hard (hopefully that day never comes).

Hope that helps!

However since late summer I’ve been seeing a badge of new notification but there was no way to clear it. I was hoping it’d go away by itself one day. I even tried the GitHub app on the phone. Nothing.

My OCD is unfortunately too big and it got to the point where I was ready to open a ticket and complain about it. But I must say they did a good job suggesting issues that might be related.

I click on the first one and quickly discover a pleasant idea - to use the API to mark all notifications as read.

Ref: https://github.com/orgs/community/discussions/6874#discussioncomment-2859125

Let’s try. I open my terminal in Warp, setup a GitHub token using 1password (I already use that) and make the cURL call.

TOKEN=$(op read op://Personal/GithubAccountItem/token)
curl -X PUT \
    -H "Accept: application/vnd.github.v3+json" \
    -H "Authorization: token $TOKEN" \
     https://api.github.com/notifications \
    -d '{"last_read_at":"2025-10-10T00:00:00Z"}'

Yayy, I feel relieved! I am no longer distracted and that matters a ton to me.

I work with a project that cannot afford any downtime. It powers a business spanning time zones around the clock. That means no cutting corners to migrate legacy, no cheat codes that I pretty regularly execute on my side projects.

I started writing this article out of frustration ignited when working on a particular task. The objective was to expose data on the new system, shown in the legacy system, that has its own unique database design, that we are forced to query from the new system running with Doctrine.

Confusing Pivot Naming to Foreign Tables/Columns

For example, one pain point is obvious looking at this table architecture:

SELECT * FROM products WHERE id = 151; -- main table
SELECT * FROM products_info WHERE product = 151 AND deleted IS NULL; -- versioning table

I really hate that the Doctrine project I'm working on uses this naming column convention, where relationship columns don't include the _id suffix, where id is the column name on the foreign table that the pivot references.

I grew up working with Laravel and then spreading into other projects. I don't need to use raw SQL as often, since typically ORMs such as Laravel's Eloquent or Drizzle from the TypeScript world are simply amazing.

I've seen a lot of open source projects (obviously not as aged) and never came across this convention. I might be completely wrong but this is so hard to get used to. Even though it's been months that I've been working with this database, I still end up wasting hours, needing another day, another look to actually SEE "oh wait, this is actually the foreign column for this table and this column."

I could imagine that this is very much MY problem and the way I was raised. I imagine if I went to that one country where people bow to say NO and turn their head to say YES, I'd want to hang myself alive :-). It is definitely difficult to unlearn a convention. I don't want to either. In my opinion, a column containing data should be self-explanatory and a column containing a reference should also be self-explanatory as to which column it links to.

No (Unique) Indices Besides the Primary Key Index

That brings me to the next part. I have absolutely no idea why back in the day, back in early 2000s/2010s, there was no convention to add unique indices/constraints.

There are so many tables with the only index being a Primary Key constraint. Just as there are many tables where duplicate data is inserted sometimes as many as a dozen times due to lack of unique constraints (and no code-side checks).

It makes me think that a career on UpWork to optimize one's database queries by merely adding indices could be a great one. Of course, that implicitly comes with a great clean-up task before constraints can be added.

Not Storing File Upload Status on the Table

This is a very recent task that I find rather comical. Some data comes along with media attached to it. A user can upload, delete, view it. I am really glad the system leverages S3 storage, since it's one of my favorite technologies in web dev. However, instead of storing the bucket path in the database, or at least storing a boolean whether the upload was done or not (it's optional and rare), the code will list all files (however it won't iterate, so it was capped at 1000 first results) in the bucket, store them into an in-memory array and then check based on a certain key if the file exists, which will decide if it's included as a pre-signed URL in the API response.

I bet that's what AI would vibe code, or what me as a 16-year-old learning to code would write. But not what a professional developer in a multinational company should ever put into production.

There are a couple of problems:

• At the very least, it should iterate and ensure all results are returned.

• It should implement a very primitive cache in Redis with tags to store that in-memory array. Tags are to be purged upon new upload. Or simply update the list in Redis upon upload.

• But that is redundant. There is no need to query S3 server-side other than uploading or deleting the file.

• Ideally, a user should store the bucket path on the table column that could be nullable.

• If a record exists, presign the URL, and return it.

• Very fast, clean, and scalable.

And there is another list of countless examples that really frustrates me. Sometimes it makes me question and self-doubt what the meaning of working on things like that is. That is hard, because I won't know until I'm gray and old if I was right or wrong thinking the way I am. But right now, I strive to remind myself that character traits and awareness that I can grow working in contexts like this, is going to be a great reward down the path.

As The Primeagen says, if you choose the easy path now, you pay the price later. And vice versa. Or as my grandma used to say, what you learn, you don't carry on your back. It feels like I'm paying the price for someone else now but we shall find out haha! I also write this to double down on my positive mindset and show myself what I am learning.

Something I'm Learning…

• If you migrate something in your folder structure, do it all or none; so you don't end up with a folder of shared, dump, smart, new and components for storing components for a frontend.

• Prefer to stick with defaults for as long as possible, because one day, when you will want to migrate, it will be FAR easier to understand how things work now and how to migrate it - looking at the legacy frontend project running WebPack with a 100 and 1 customization.

• Isolate to one change per PR as much as possible. I hate this in practice because it feels so easy and tempting to refactor everything at once (might be my ADHD/OCD to blame) but it is Achilles' heel when working with a sensitive system and legacy code base.

• Patience, really, just be slow and do things slow. Perhaps the biggest thing that sometimes drives me absolutely nuts. I will sit in front of a computer yet my heart will race up like I'm running up a hill. It's funny I react that way. Makes me even laugh afterwards realizing the irony. But it does make me wonder how I can build this mental muscle to be indifferent. Because with the rise of "vibe coding," people with the ability to work on legacy will be in as high demand as developers were a decade ago.

• Of course, plenty of examples of what NOT to do in architecting the system for scale and many years to maintain in production.

Takeaways

• If you're not a very calm person by personality (that is, you have ADHD), meditate each morning. As little as 8 minutes can do wonders. 15 minutes is wonderful. It helps a lot to remain unaffected when literally needing to dive into challenging code. I now do this daily. And without it, I notice how I'm all over the place. Just to double down, this with running outside, crosses out more than half of the pills & tips in the /r/Nootropics subreddit.

• Consistently remind yourself that frustration comes as the by-product from learning something new. It is a chemical that gets released in our brain to enable elasticity to establish new synapses (the links between neurons, between our knowledge storage). Plus, building this versatility against legacy is going to be a superior shield against AI taking away your job sooner. I don't know if I know this for 100% sure but I'd rather believe this than waste energy doubting.

• I'm here to learn. I hope that my mind will absorb these mistakes of others and I can learn from them. So that the code I write will look better for future developers. :)

1. Rewrite drizzle schema from sqlite to PostgreSQL and generate migration (*.sql) files.

2. Setup connection using Neon WebSocket proxy and test migration and code.

3. Attempt various ways to copy and settle on COPY via CSV type using stdout and stdin.

4. Perform complete migration during not busy moment.

Why did I do this?

In the early days of my little project for citizens of the world, I made a mistake by optimizing prematurely. I had initially chosen Cloudflare’s D1 database but I quickly outgrew the limits and was forced to migrate to Turso.

I wrote about that journey in this https://lukasnotes.dk/migrating-large-d1-to-turso post.

After using the cloud option and the paid plan for a few months, I was still hitting the limits and seeing my bill grow even though there wasn’t that much usage. Without going into details, due to nature my app produces a lot of writes, which would cause a lot of changes in the WAL file, and that is how libsql tracks changes and sync to the local embedded replicated database. I do that to have zero latency reads from Laravel.

I had built a simple sync script, should you find that relevant. https://github.com/flexchar/turso-sync

The main Laravel app leverages PostgreSQL. The god level database as some address in the rise of AI. It made me double my work in certain cases and limited the type of queries I could write since data was split across several databases - Postgres and Sqlite.

After some procrastination, last November I got myself together and began tackling the challenge.

Is Turso bad? Is it not good enough?

No, it is not bad at all. It’s all about picking the right tool for the job.

It is amazing what the guys behind Turso are doing with the Sqlite’s fork, libsql. I had a chance to have a call with the very man behind, Glauber Costa as well as attend a few weekly meetings on their Discord. They’re going strong and I believe they will go far.

Let alone their recent announcement for a complete Sqlite rewrite in Rust - the project called Limbo (https://github.com/tursodatabase/limbo).

I think for standard blogs or simple CRUD apps, it’s more than enough. Don’t hesitate too much with “how but start now”.

For me, I had cloned my app from another project where it as all about latency and I kept driving the road without realizing it was the wrong path for me.

Making a plan

I had quite a challenge actually doing the “not how but start now” as I wanted the perfect option. It’s such a weakness of mine. The first problem was to pick between:

A: do I create entire new database within the same pgsql container, or
B: do I run migrations in the same database that Laravel uses and is running its own migrations.

Neither sounded too nice. Route A would prevent me from making joints across tables in both sources. Route B could potentially cause issues with migrations, backups and collide each other.

Then came the thought of Route C. Let’s use the same database but different schemas. It’s like a namespace to group tables. It’s something unique to Postgres as neither Mysql nor Sqlite has it.

There’s a nice explanation on Reddit: https://www.reddit.com/r/PostgreSQL/comments/qlqmyh/can_someone_help_me_understand_schemas_and_how/. I had never used them myself but Laravel by default uses public schema. I decided to create drizzle schema and have a logical separation for the migrations and data being written from the worker.

It felt fine with me, and in retro, I can still say, it was a good decision.

Establishing a connection to self-hosted Postgres from Cloudflare Worker

Before copying the data, I updated my worker’s Drizzle config to connect to Postgres. It didn’t come without challenges.

I care very much about the security and privacy, and I would really like to live my life without ending up leaking anyone’s data. I also self-host entire stack as it’s absurdly cheap on Hetzner. Even if one was to SSH on the server, the database connection is not exposed. There are no ports and TCPs. I use Docker and leverage UNIX sockets for ultimate performance. I absolutely love that!

Anyways, my ideal solution was to use Cloudflare Tunnel to tap into the container and expose the database on internal domain (protected by Cloudflare Zero Access) to my worker, also running on CF.

The issue is that worker’s runtime and postgres-js package did not support connecting directly. I could get some things to work locally but then not on the cloud/production, or other way around.

Fortunately, Neon Tech (the serverless & bottomless storage postgres startup) has a wonderful container to proxy requests from serverless environments to any postgres instance. However as I mentioned before, I prefer to use UNIX sockets. I happened to get very lucky to mix bits of Go knowledge with Claude Sonnet and have an updated code to connect via socket.

So now I had a docker compose stack with pgsql, neon-ws-proxy and cloudflare-tunnel exposing under private domain. After tinkering a bit with Drizzle Neon Client, I was able to connect. One catch, I have to close connections myself as sometimes it’d get issues.

Boom, it felt very nice. I was able to run migrations and had a list of tables in drizzle schema, right in my Laravel’s existing database.

Copying the data. Slow start, fast finish.

I have users using almost all day long across the globe. Even though I can afford a bit of data loss due its nature in this table, I really want that to be as close to zero rows as possible. That means, I want to migrate from production to my local PG. Check everything runs smoothly, if everything seems & feels good, deploy my worker and start migrating ASAP while new requests are written to the new database and some will temporarily fail due otherwise missing data.

Via script to read and insert (failed route)

Initially I ought to copy via writing simple script in TS and using drizzle to read from Turso connection and insert into PostgrSQL connection. In short, I’ve wasted a good day and it was a pure failure. It was too slow.

It seems that for whatever the reason, Turso reads are slow, and if reading a table with bigger rows, I’d quickly exceed the maximum 10MB response size. A table of several millions of rows but I can only query a few hundred at most per time. That went slow. To put into perspective, I had around 30GBs of data and the migration could take entire week. I still don’t know why it was that slow. But this is also one of the reasons why I wanted to migrate. Sometimes abnormal things would occur with LibSQL instance and the Cloudflare sitting in front of it.

Since I was self-hosting libsql for around 4 months before the migration, I established a local connection forwarding the port from the server to my laptop, then I used Turso CLI turso db shell http://localhost:8080?token={authTokenHere} to access the shell.

Via .dump and COPY (success)

I then searched for possible tools to automagically import sql from Sqlite into Postgres. While there are several options, none gave me the magic and I came across the copy command and the csv route.

I dumped entire database to the datalake.db file using turso cli.

After a bit of trial and error, I found that dumping a single table to CSV file, copying it to the pgsql container and then loading via PG’s COPY statement I could import millions of rows in a matter of 1-3 minutes.

I had few gotchas.

1. I had to make sure that the table is completely empty before running the copy, or I would encounter unique constraint issues or sometimes some errors.

2. Next gotcha was converting the date time string from Sqlite’s milliseconds to a proper UTC string that PG could understand.

3. Finally, my source schema was built over time with a lot of incremental migrations adding new columns, however when I rewrote it for PG, I only had one migration with a bit different order of columns. Making sure the order of columns matches exactly in the source and target.

Eventually I nailed this process to the entire one-liner.

time sqlite3 /turso/datalake.db -header -csv "SELECT col_1,col_2,another_name,DATETIME(created_at/1000, 'unixepoch') as created_at FROM turso_table;" | PGPASSWORD=$DB_PASSWORD psql -U $DB_USERNAME -d $DB_DATABASE -c "\COPY drizzle.turso_table (my_col_1,my_col_2,my_another_col,created_at) FROM STDIN WITH (FORMAT csv, HEADER true)"

If you’re going to do the same, please read this carefully and play around to understand how it works. It gives far more value than a blind copy-paste.

Execution

That brings to the final step. I had successfully COPY-ied data to PostgreSQL. I have established connection from Drizzle in the Worker as well as able to read using Laravel Eloquent models by updating $table property to include the schema prefix.

I’m very happy!

I tweak my note file with copy paste commands ready for each database, I ensure I can access my production databases and containers.

I reflect on the usage logs and pick the least busy hour that happens to be around 7 AM CET and execute it.

It goes like:

1. Complete dump of fresh production database into an sqlite file.

2. Create fresh migration on Production and deploy the worker.

3. Dump each table as CSV from the sqlite file.

4. COPY the CSV into Postgres table, starting with most busy ones to the least ones.

5. Rinse & repeat until I’m done.

6. Deploy changes on Laravel app.

7. Enjoy!

Runtime adjustments.

When using Turso driver with Drizzle, I often would make a lot of calls in an executionContext.waitUntil() method, that essentially awaits promises after the worker has sent HTTP response.

Using PostgreSQL Neon Proxy driver, I quickly began facing Too Many Connections ocean of errors. I had to rewrite my endpoints to await before response and then explicitly close the connection from Neon driver.

Since lots of these endpoints are write only (think analytics type of API), it was rather un-impactful change to the end user. It did require me to do a bit of rewrite to have a better access to the DB client. I’m using HonoJS and its c.set(‘db’, databaseClient) is a wonderful way to pass it around.

Final thoughts. Was it worth?

Absolutely!

I wish I had done this from the get-go. Although back when I first started, some things weren’t available so I probably would have struggled more.

My greatest wins are

1. No more time outs or errors when SELECT'ing too much data. Sqlite cannot leverage multiple indices for faster queries, where as Postgres does it much faster.

   1. Everyone says Sqlite is the fastest at read speeds, so I wouldn’t rule out a skill issue here.

2. Much less code to maintain!! Very important win. One less database to backup (and restore, should that ever happen).

3. I can write more complex and interesting queries. This is especially valuable for analytics. It has been an enormous pleasure.

Pre history

When I first registered, I was using a bank that supported holding balances in both DKK, EUR and USD. That made it a pain to do proper book keeping because of mixed currencies. I tried Dinero.dk which was nice but it freaked out when I tried setting currencies other than Danish Krone. Their support referred me to VISMA, the more serious platform. However I quickly found that it was prohibitively complicated and demanding for my very simple case.

As time proven, I settled for an Excel sheet. Basic columns such as total paid, vat amount, currency etc. Then I did my first submission of VAT, it is called Moms in Danish. I made some formulas and got what I wanted. But it was too much hassle than I would have preferred. I did it again and with the turn of 2024 I knew I will automate this as every real developer would do.

Code

Something that changed is that this year, 2024, I decided to pay using Danish Krones no matter the destination currency (aka, to leverage the perks of credit cards). This removed the previous challenge and made it even simpler. Then I wrote the following code to fetch the csv of my records in Google Sheets and calculate.

More fluff of mine

Ironically enough, even though I am better versed in English than in Danish at the time being, I find certain explanations in Danish easier to understand. Speaking of which, I got to really acknowledge how great Denmark is for making it easy to understand the taxes.

Both the documentation on the website provided in multiple languages and the helpful phone line, is what makes me pay taxes with the smile. Seriously good job by SKAT and that's one of places Denmark really stands out.

Finally the code

So all in all, this is the script I came up with.

import { parse } from 'csv-parse/sync';

const { SHEETS_CSV_URL, FROM_DATE, TO_DATE } = Bun.env;

if (!SHEETS_CSV_URL) {
    console.error('SHEETS_CSV_URL is required');
    process.exit(1);
}
if (!FROM_DATE) {
    console.error('FROM_DATE is required');
    process.exit(1);
}
if (!TO_DATE) {
    console.error('TO_DATE is required');
    process.exit(1);
}

const csvString = await (await fetch(SHEETS_CSV_URL)).text();

const records = parse(csvString, {
    cast: true,
    cast_date: true,
    columns: true,
    skip_empty_lines: true,
});

type Record = {
    invoiceId: string;
    date: Date;
    name: string;
    type: 'A - Services' | 'A - Goods' | 'B - Services';
    inEu: boolean;
    inDk: boolean;
    grandTotal: number;
    vatRate: number;
    baseValue: number;
    vatValue: number;
};

// filter records from January 1st, 2024 to March 31st, 2024
const filteredRecords = records.filter((r: Record) => {
    const date = r.date;
    return date >= new Date(FROM_DATE) && date <= new Date(TO_DATE);
});

// console.log(filteredRecords);
// process.exit(0);

// Caclulate
// https://skat.dk/en-us/businesses/vat/vat-on-international-trade/reporting-your-international-trade
// https://skat.dk/erhverv/moms/moms-ved-handel-med-udlandet/indberet-din-handel-med-udlandet
const tax = {
    'vat-in-dk': 0, // Moms i DK til Købsmoms (Input VAT) (VAT deductible)

    // VAT on goods purchased outside Denmark (both the EU and third countries).
    'vat-on-goods-purchased-outside-denmark': 0,
    // VAT on services purchased outside Denmark subject to a reverse charge
    'vat-on-services-purchased-outside-denmark-subject-to-a-reverse-charge': 0,
    // to calculate Købsmoms (Input VAT) (VAT deductible)
    'vat-on-services-purchased-outside-denmark-outside-eu': 0,

    'eu-sales-with-vat': 0,
    'eu-sales-without-vat': 0,

    'vat-paid': 0, // Købsmoms
    'vat-collected': 0, // Salgsmoms

    // Boxes (base value here excl. VAT)
    'box-a-services': 0,
    'box-a-goods': 0,
    // Not relevant for my business
    'box-b-services': 0,
    'box-b-goods': 0,
    'box-c-services': 0,
};

tax['vat-in-dk'] = filteredRecords
    .filter((r: Record) => r.inDk)
    .filter((r: Record) => r.type === 'A - Services' || r.type === 'A - Goods')
    .reduce((acc: number, r: Record) => acc + r.vatValue, 0);

let vatOf25estimatedForReverseCharge = 0;

// Moms af varekøb i udlandet (både EU og lande uden for EU)
// VAT on goods purchased outside Denmark (both the EU and third countries).
// Enter the VAT payable on services purchased outside Denmark during the VAT period (both EU countries and third countries).
// You calculate the VAT as 25% of the invoice value of the services purchased during the period.
tax['vat-on-goods-purchased-outside-denmark'] = filteredRecords
    .filter((r: Record) => !r.inDk)
    .filter((r: Record) => r.type === 'A - Goods')
    .reduce((acc: number, r: Record) => {
        let value = r.vatValue;

        if (r.vatRate === 0) {
            value = r.grandTotal * 0.25;
            vatOf25estimatedForReverseCharge += value;
        }

        return acc + value;
    }, 0);

// Moms af ydelseskøb i udlandet med omvendt betalingspligt
// VAT on services purchased outside Denmark subject to a reverse charge
tax['vat-on-services-purchased-outside-denmark-subject-to-a-reverse-charge'] =
    filteredRecords
        .filter((r: Record) => !r.inDk && r.inEu)
        // Only report the value of the EU services you purchased in box A - services.
        .filter((r: Record) => r.type === 'A - Services')
        .filter((r: Record) => r.vatRate === 0)
        .reduce((acc: number, r: Record) => {
            let value = r.vatValue;

            if (r.vatRate === 0) {
                value = r.grandTotal * 0.25;
                vatOf25estimatedForReverseCharge += value;
            }

            return acc + value;
        }, 0);

tax['vat-on-services-purchased-outside-denmark-outside-eu'] = filteredRecords
    .filter((r: Record) => !r.inDk && !r.inEu)
    .filter((r: Record) => r.type === 'A - Services')
    .reduce((acc: number, r: Record) => {
        let value = r.vatValue;

        if (r.vatRate === 0) {
            value = r.grandTotal * 0.25;
            vatOf25estimatedForReverseCharge += value;
        }

        return acc + value;
    }, 0);

// Print the manually added 25% on reverse charge
console.info({ vatOf25estimatedForReverseCharge });

// Købsmoms // Input VAT (VAT deductible)
// You may include amounts from the following fields:
// - VAT on goods purchased outside Denmark.
// - VAT on services purchased outside Denmark subject to a reverse charge.
tax['vat-paid'] =
    tax['vat-in-dk'] +
    tax['vat-on-goods-purchased-outside-denmark'] +
    tax[
        'vat-on-services-purchased-outside-denmark-subject-to-a-reverse-charge'
    ] +
    tax['vat-on-services-purchased-outside-denmark-outside-eu'];

// Box A - goods
// VAT on goods purchased outside Denmark (both the EU and third countries).
// You should report the value of your purchase of goods from other EU countries in box A - ‘goods’  on your VAT return.
// KISS: Hardware that I bought from the EU but not in Denmark
// Rubrik A - varer. Værdien uden moms af varekøb i andre EU-lande - EU-erhvervelser.
tax['box-a-goods'] = filteredRecords
    .filter((r: Record) => r.inEu && !r.inDk)
    .filter((r: Record) => r.type === 'A - Goods')
    .reduce((acc: number, r: Record) => acc + r.grandTotal, 0);

// Rubrik A - ydelser. Værdien uden moms af ydelseskøb i andre EU-lande.
// Box A - services
// You should report the value of your purchase of services from other EU countries in box A - ‘services’  on your VAT return.
// KISS: Services that I bought from the EU but not in Denmark, the base value w/o VAT
tax['box-a-services'] = filteredRecords
    .filter((r: Record) => r.inEu && !r.inDk)
    .filter((r: Record) => r.type === 'A - Services')
    .reduce((acc: number, r: Record) => acc + r.grandTotal, 0);

// Box B - services
// The value of certain sales of services exclusive of VAT to other EU countries. To be reported under ‘EU-salg uden moms’ (EU sales exclusive of VAT)
// Rubrik B-ydelser. Værdien af visse ydelsessalg uden moms til andre EU-lande. Skal også indberettes til systemet "EU-salg uden moms".
// KISS: Services that I sold to the EU without charging VAT (reverse charge) (typically B2B)
tax['box-b-services'] = filteredRecords
    .filter((r: Record) => r.inEu && !r.inDk)
    .filter((r: Record) => r.type === 'B - Services')
    .reduce((acc: number, r: Record) => acc + r.grandTotal, 0);

// EU sales with VAT
// The value of certain sales of goods and services exclusive of VAT to other EU countries. To be reported under ‘EU-salg med moms’ (EU sales with VAT)
// KISS: Sales of goods and services to the EU where I charged VAT (typically B2C)
tax['eu-sales-with-vat'] = filteredRecords
    .filter((r: Record) => r.inEu && !r.inDk)
    .filter((r: Record) => r.type === 'B - Services')
    .filter((r: Record) => r.vatRate > 0)
    .reduce((acc: number, r: Record) => acc + r.grandTotal, 0);

// EU sales without VAT
// The value of certain sales of goods and services exclusive of VAT to other EU countries. To be reported under ‘EU-salg uden moms’ (EU sales exclusive of VAT)
tax['eu-sales-without-vat'] = filteredRecords
    .filter((r: Record) => r.inEu && !r.inDk)
    .filter((r: Record) => r.type === 'B - Services')
    .filter((r: Record) => r.vatRate === 0)
    .reduce((acc: number, r: Record) => acc + r.baseValue, 0);

// No sales for me... yet :)
// console.log(
//     `
// Report the value of your sale in two different places in E-tax for businesses:

// In your VAT return within the normal deadlines that apply to your business.
// Under ’EU-salg uden moms’ (EU sales exclusive of VAT) by the 25th day of each month.

//     `.trim(),
// );

// Round to no decimal places
Object.keys(tax).forEach((key) => {
    tax[key] = Math.round(tax[key]);
});

// Add Danish labels, it is easier to know where to put the numbers in the tax form
const dansk = {
    'Moms af varekøb i udlandet (både EU og lande uden for EU)':
        tax['vat-on-goods-purchased-outside-denmark'],
    'Moms af ydelseskøb i udlandet med omvendt betalingspligt':
        tax[
            'vat-on-services-purchased-outside-denmark-subject-to-a-reverse-charge'
        ],
    Købsmoms: tax['vat-paid'],
    'Rubrik A - varer': tax['box-a-goods'],
    'Rubrik A - ydelser': tax['box-a-services'],
    'Rubrik B - ydelser': tax['box-b-services'],
    'EU-salg med moms': tax['eu-sales-with-vat'],
    'EU-salg uden moms': tax['eu-sales-without-vat'],
};

console.dir(dansk);

I like TypeScript for how easy it is throw something together and just run the code. I like Bun that it allows me to execute TypeScript with no need to transpile first.

This script will print all the numbers that I need to fill out in the SKAT form and that is soo so nice. If you're doing the same, you're welcome to use it too. Just make sure to match the columns.

On GitHub

I also put it up in the repo here https://github.com/flexchar/bun-scripts/tree/main/skat. I haven't figured out the ultimate structure for these seldom used scripts.

PS. If you notice that something is not quite right, please do let me know. At the end of the day, it is a little challenge to interpret the taxation rules. :)

I came across Modal last summer when I was on a self-inspired mission to run BLOOM 176B model as an open-source competition to ChatGPT.

The guys at modal were amazing and after several days of tinkering after my work, I got it up using 6 GPUs simultaneously. I am not a big fan of Python but it is essentially the only way for ML computing and the way Modal allows to run the code on the cloud is just lovely.

Fun fact, one of the founders is from Stockholm. There's a very interesting podcast where he shares his journey about how internet was back then in Sweden. I don't have link right now but it's worth a search.

Recently I wanted to fine tune gpt 3.5 but my inputs got flagged. I have a big dataset so instead of going through their free moderation API I felt it was the perfect push to try Unsloth. The guys are absolute geniuses that wrote their own kernels.

By the way, Modal gives everyone free 30usd/mon worth of credits so you get plenty of room for trial & error.

Fine tunning Llama 3 70B

The primary audience for Unsloth are everyone running their trials on Google Collab. That is something that I never got too comfortable with, I like the code on my own laptop in my own git repo.

It was a bit of challenge to figure out the right libraries and the base docker image to setup to run the library. Fortunately it didn't take too long diving in their discord to come across the one, and the rest of the code was born.

It took me 10 minutes to fine tune quantized Llama 3 70b 4bit using Modal's H100. I wonder how long it would take had I access to Unsloth's premium code supporting multiple GPUs. Anyway, super nice piece of code. As a developer, I love to have a little code as possible. You're welcome to use it.

Code

Remember to read the part below too.

from datetime import datetime
import secrets
import os
from modal import App, Secret, Image, Volume, gpu
from pathlib import PurePosixPath


HUGGING_FACE_USERNAME = "name"

app = App("unsloth-experiment")

# Volumes for pre-trained models and training runs.
pretrained_volume = Volume.from_name("unsloth-pretrained-vol", create_if_missing=True)
runs_volume = Volume.from_name("unsloth-runs-vol", create_if_missing=True)
VOLUME_CONFIG: dict[str | PurePosixPath, Volume] = {
    "/pretrained": pretrained_volume,
    "/runs": runs_volume,
}


image = (
    Image.from_registry("pytorch/pytorch:2.2.0-cuda12.1-cudnn8-devel")
    .apt_install("git")
    .pip_install(
        "torch==2.2.1",
    )
    .run_commands(
        'pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"',
    )
    .run_commands(
        "pip install --no-deps packaging ninja einops flash-attn xformers trl peft accelerate bitsandbytes",
    )
    .pip_install("huggingface_hub", "hf-transfer", "wandb")
    .env(
        dict(
            HUGGINGFACE_HUB_CACHE="/pretrained",
            HF_HUB_ENABLE_HF_TRANSFER="1",
        )
    )
)


@app.function(
    image=image,
    gpu=gpu.H100(),
    volumes=VOLUME_CONFIG,
    timeout=3600 * 1,
    _allow_background_volume_commits=True,
    secrets=[Secret.from_name("wandb-secret")],
)
def train(data_raw: str, model_name: str, run_name: str, run_folder: str):
    from unsloth import FastLanguageModel, chat_templates
    import torch
    from trl import SFTTrainer
    from transformers import TrainingArguments, EarlyStoppingCallback
    from datasets import load_dataset
    import wandb

    # Track in WanDB
    wandb.init(project=app.name, name=run_name, tags=["unsloth", "H100", model_name])

    # Write the dataset to a file
    file_path = "dataset.jsonl"
    with open(file_path, "w") as data_file:
        data_file.write(data_raw)

    print(f"Loading dataset from {file_path}...")
    dataset = load_dataset("json", data_files={"train": file_path}, split="train")

    # Print number of examples
    print(f"Number of examples: {len(dataset)}")

    # Load Llama model
    max_seq_length = 4096
    print(f"Initalizing model with max_seq_length={max_seq_length}...")
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name=model_name,
        max_seq_length=max_seq_length,  # Supports RoPE Scaling interally, so choose any!
        dtype=None,  # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
        load_in_4bit=True,  # Use 4bit quantization to reduce memory usage. Can be False.
    )

    # Do model patching and add fast LoRA weights
    model = FastLanguageModel.get_peft_model(
        model,
        r=16,  # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
        target_modules=[
            "q_proj",
            "k_proj",
            "v_proj",
            "o_proj",
            "gate_proj",
            "up_proj",
            "down_proj",
        ],
        lora_alpha=16,
        lora_dropout=0,  # Supports any, but = 0 is optimized
        bias="none",  # Supports any, but = "none" is optimized
        # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
        use_gradient_checkpointing="unsloth",  # True or "unsloth" for very long context
        random_state=3407,
        max_seq_length=max_seq_length,
        use_rslora=False,  # We support rank stabilized LoRA
        loftq_config=None,  # And LoftQ
    )

    # https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2#scrollTo=LjY75GoYUCB8&line=1&uniqifier=1
    # ChatML is the default chat template
    tokenizer = chat_templates.get_chat_template(tokenizer)

    def formatting_prompts_func(entry):
        # Example of the payload
        # print(entry["messages"])
        convos = entry["messages"]
        texts = []
        for convo in convos:
            text = tokenizer.apply_chat_template(
                convo, tokenize=False, add_generation_prompt=False
            )
            texts.append(text)
        return {
            "text": texts,
        }

    print(f"Mapping prompts...")
    dataset = dataset.map(formatting_prompts_func, batched=True)

    # Print number of examples
    print(f"Number of examples: {len(dataset)}")

    trainer = SFTTrainer(
        model=model,
        train_dataset=dataset,
        dataset_text_field="text",
        max_seq_length=max_seq_length,
        tokenizer=tokenizer,
        args=TrainingArguments(
            per_device_train_batch_size=2,
            gradient_accumulation_steps=8,
            warmup_steps=10,
            max_steps=30,
            fp16=not torch.cuda.is_bf16_supported(),
            bf16=torch.cuda.is_bf16_supported(),
            logging_steps=1,
            output_dir=run_folder,
            optim="adamw_8bit",
            seed=3407,
            run_name=run_name,
            report_to="wandb",
            # load_best_model_at_end=True,  # Load the best model in terms of metric at the end of training
            # metric_for_best_model="train_loss",  # Specify the metric to use for early stopping
            # greater_is_better=False,  # Specify if higher values are better for the specified metric
        ),
        # callbacks=[
        #     # Stop training if eval loss does not improve for 3 checkpoints
        #     EarlyStoppingCallback(3),
        # ],
    )

    print(
        f"Starting training run with {torch.cuda.device_count()} {torch.cuda.get_device_name()}"
    )
    trainer.train()
    model.save_pretrained(f"{run_folder}/lora_model")  # Local saving

    VOLUME_CONFIG["/runs"].commit()

    return run_name


@app.function(
    image=image,
    volumes=VOLUME_CONFIG,
    _allow_background_volume_commits=True,
    secrets=[Secret.from_name("my-huggingface-secret")],
)
def online(data_raw: str, model_name: str, train_suffic: str):
    from huggingface_hub import snapshot_download

    # Ensure the base model is downloaded
    try:
        snapshot_download(model_name, local_files_only=True)
        print(f"Volume contains {model_name}.")
    except FileNotFoundError:
        print(f"Downloading {model_name} ...")
        snapshot_download(
            model_name,
            token=os.environ["HF_TOKEN"],
        )

        print("Committing /pretrained directory (no progress bar) ...")
        VOLUME_CONFIG["/pretrained"].commit()

    # Write config and data into a training subfolder.
    time_string = datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
    run_name = f"unsloth-{time_string}"
    # if train_suffic is not empty, append it to the run_name
    if train_suffic := train_suffic.strip():
        run_name += f"-{train_suffic}"
    run_folder = f"/runs/{run_name}"
    os.makedirs(run_folder)
    print(f"Preparing training run in {run_folder}.")

    # Start training run.
    print("Spawning container for training.")
    train_handle = train.spawn(data_raw, model_name, run_name, run_folder)
    with open(f"{run_folder}/logs.txt", "w") as f:
        msg = f"train: https://modal.com/logs/call/{train_handle.object_id}"
        f.write(msg)
        print(msg)
    VOLUME_CONFIG["/runs"].commit()

    return run_name, train_handle


def run_cmd(cmd: str, run_folder: str):
    import subprocess

    # Ensure volumes contain latest files.
    VOLUME_CONFIG["/pretrained"].reload()
    VOLUME_CONFIG["/runs"].reload()

    # Propagate errors from subprocess.
    if exit_code := subprocess.call(cmd.split(), cwd=run_folder):
        exit(exit_code)

    # Commit writes to volume.
    VOLUME_CONFIG["/runs"].commit()


@app.function(
    image=image,
    volumes=VOLUME_CONFIG,
    secrets=[Secret.from_name("huggingface-secret-write")],
)
def upload_to_hf(run_folder: str, name: str):

    hf_token = os.environ["HF_TOKEN"]

    CMD = f"huggingface-cli upload --token {hf_token} --private {HUGGING_FACE_USERNAME}/{name} lora_model"
    # To Hugging Face Hub
    run_cmd(CMD, run_folder)

    # ToDo: upload to S3 for Predibase


# ChatML dataset
@app.local_entrypoint()
def main(
    data: str,
    train_suffic: str,
    model_name: str = "unsloth/llama-3-70b-bnb-4bit",
):
    # Read data file and pass contents to the remote function.
    with open(data, "r") as dat:
        run_name, train_handle = online.remote(dat.read(), model_name, train_suffic)

    # # Write a local reference to the location on the remote volume with the run
    with open(".last_run_name", "w") as f:
        f.write(run_name)

    # Wait for the training run to finish.
    train_handle.get()
    print(f"Training complete. Run tag: {run_name}")
    print(
        f"To inspect weights, run `modal volume ls unsloth-runs-vol {run_name}/lora_model`"
    )

    print(f"Uploading to Hugging Face Hub...")
    run_folder = f"/runs/{run_name}"
    upload_to_hf.remote(run_folder, run_name)

To use, you will need to set:

• huggingface-secret-write to upload the adapter to Hugging Face, or delete that part of the code.

• wandb-secret to track the fine tunning progress in WanDB, or delete the code, which I would highly advise against because you really do want to know how it's doing.

Finally prepare OpenAI type of dataset and launch training using the following command: modal run --detach learn.py --data dataset.jsonl --train-suffic cph

Observe the training eval/loss, over fitting and over training yields no value and I haven't had a chance to implement the EarlyStopping yet.

Bonus Tips

For a good fine tunning:

• make sure to isolate to a single problem solving, no "do it all" type of stuff;

• start with smallest model and then go up;

• Mistral 7B and Llama 8B are some of the state-of-art choices available for free;

• if you need chat, stick to ChatML template - it will save you a lot of trouble down the road;

• prefer base model (not instruct) to have a fresh starting point as possible, or if picking a community tuned model such as Eric Hartford's - make sure you don't mix chat templates (given you train a chat model);

• quality of dataset is everything, few shitty examples would ruin as powerful model as GPT 4 and only high quality examples would make small models outperform GPT 4 (search for Predibase adapters case study on this);

Enjoy!

If you end up training something fun, let me know!

If you're coming from google because you saw unknown variable 'default-auth=mysql_native_password'. - all you need to do is to change --default-authentication-plugin=mysql_native_password to --mysql-native-password=ON.

Story

I recently did a deployment where a docker compose project had own mysql that was only pinned to the version mysql:8. The config snippet below.

mysql:
    image: mysql:8
    restart: always
    command: --default-authentication-plugin=mysql_native_password
    volumes:
        - db_data:/var/lib/mysql
        - db_socket:/var/run/mysqld
    environment:
        - MYSQL_DATABASE=xxxxx
        - MYSQL_ROOT_PASSWORD=xxxxx
    healthcheck:
        test: ['CMD', 'mysqladmin', 'status', '-hlocalhost', '-uroot', '-pxxxxxx']
        interval: 1m
        timeout: 5s
        retries: 3
        start_period: 1m

Immediately one could point out that it's a huge mistake to only pin by a major version. For a truly production project, I would almost never host my own database - aka, I would always use a managed one.

Anyhow, this one slipped through and caused completely unnecessary downtime.

Debugging

After looking through the looks I could see mysql 8.4.0 Plugin 'mysql_native_password' is not loaded and unknown variable 'default-auth=mysql_native_password' lines and that the server is shutting down.

Almighty Google!

That means I started surfing. Since it was released just two days ago (on April 30th), there weren't any immediately results that seemed relevant as deemed by my brains.

Based on that, I searched for change log/release log. I land on this page https://dev.mysql.com/doc/relnotes/mysql/8.4/en/news-8-4-0.html and do Cmd + F (search in page) to see if there has been anything regarding the plugin.

Boom, lucky result under "Deprecation and Removal Notes" section

Important Change: The deprecated mysql_native_password authentication plugin is now disabled by default. It can be enabled by starting MySQL with the new --mysql-native-password=ON server option, or by adding mysql_native_password=ON to the [mysqld] section of your MySQL configuration file.

That led me to update the compose config and fix the downtime.

mysql:
    image: mysql:8.4
    restart: always
    command: --mysql-native-password=ON
    volumes:
        - db_data:/var/lib/mysql
        - db_socket:/var/run/mysqld
    environment:
    #....

That's it. I hope by writing this I won't land on this mistake again. Restoring from backup is such as boring thing to do.

Or maybe it's a sign to stop procrastinating migration to pg... would be lovely to leverage Cloudflare's hypertunnel...

History

Two weeks ago I found myself sharing my side project where I transcribe a whole dozen of videos into a list of JSON files containing word-level transcripts. Everyone who has been into the AI news since end of 2022 will probably know about Whisper. It's one of the wildest value drops for everyone.

Solving GPU (for Whisper) runtime

However running Whisper model locally can get quite demanding, can be slow or impossible given on the hardware. I am a big fan of high accuracy, high value and high speed. That means I want the large model and low speed.

Initially I have used Modal.com for running my Python code. It worked well but I don't want to maintain python. I want to be as simple as possible. In winter I came across Replicate.com (there's at least half a hundred of hosted providers like that, I'm trying to keep my bookmarks updated). It offers turian/insanely-fast-whisper-with-video, which in short is much faster version Whisper runtime. OpenAI tends to provide a working but not optimised code.

Solving file hosting

It didn't take long for me to try it using cURL and jq to have the transcript. But there was one problem. Many of the files are on my laptop. Such as a meeting I've just had at Flügger. I need it available on some url. There are data protections and again, I wanted to keep as simple as possible. One thing that we all love and use is S3 storage. Lucky me, I came to notice that Cloudflare's R2 (their S3 offering) has finally added the support.

So I created a very convenient bucket called tmp and a few rules to delete files after one day if path prefix matches s3://tmp/1d/ . Boom. Now I can use CLI to copy paste the file into private bucket that cleanses itself and get a signed URL, that is publicly accessible and private.

Reusable script

My terminal, Warp.dev, after updating itself popped up with a new notebook announcement. I must have gotten lucky again. Instead of writing a shell script, now I could create a Python like notebook that I can add comments and remind my future self what the heck I'm doing here. Sometimes it's scary how quickly I can forget certain commands and how they work, if I haven't thought about them for a few weeks.

The code

You may download this code as a Warp notebook from here:
https://gist.github.com/flexchar/0a9c6ecf0c1b9c2592e45af78c30cb23

or right in your holy shell

wget https://gist.githubusercontent.com/flexchar/0a9c6ecf0c1b9c2592e45af78c30cb23/raw -O video-to-json.md

Prerequisites

• This script uses s5cmd as an s3 client which can be installed using brew install s5cmd, or equivalent on other systems.

• This script assumes you have r2 alias set for

r2='s5cmd --credentials-file ~/.config/s3/r2-config.cfg --endpoint-url https://{your bucket id}.r2.cloudflarestorage.com'

• Replicate token set as an environment variable

export REPLICATE_API_TOKEN=

Process

1. Upload video to S3 on Cloudflare R2 bucket. This will create as web accessible link so we don't have to kill ourselves uploading the file directly. The bucket files are not browsable by public so they're still private.

S3_PATH="s3://tmp/7d/{{destination_name}}"
r2 cp --sp {{source_path}} $S3_PATH

Couple notes: Our tmp bucket is configured with two rules that delete files that start with either 1d and 7d prefix respectively. The rule of course matches the prefix duration. As the bucket name implies, this bucket shouldn't be used for anything that should be persistent. Each file is wiped on its 30 days year old birthday, aka, it gets removed. Please see section Object lifecycle rules in your bucket settings.

2. After uploading, get a publicly accesible link - that is presign the URL.

SIGNED_URL=$(r2 presign --expire 1h $S3_PATH)
echo $SIGNED_URL

3. Submit to Replicate

REPLICATE_RES=$(curl --location 'https://api.replicate.com/v1/predictions' \
--header 'Content-Type: application/json' \
--header "Authorization: Token $REPLICATE_API_TOKEN" \
--data @- <<EOF
{
    "version": "4f41e90243af171da918f04da3e526b2c247065583ea9b757f2071f573965408",
    "input": {
        "url": "$SIGNED_URL",
        "task": "transcribe",
        "timestamp": "chunk",
        "batch_size": 64,
        "language": "en"
    }
}
EOF
)
echo $REPLICATE_RES | jq
WHISPER_ID=$(echo $REPLICATE_RES | jq -r '.id')
echo $WHISPER_ID

This is a cold-started ML endpoint. It may take up to 3 minutes until your transcript is ready.

You could completely add a loop that calls endpoint and checks for status, every 5 seconds, make sure to not DDoS them.

4. Get the results using the id in the response

OUTPUT_FILE="{{output_filename}}.json"
echo $OUTPUT_FILE
curl --location "https://api.replicate.com/v1/predictions/$WHISPER_ID" \
--header "Authorization: Token $REPLICATE_API_TOKEN" | jq > $OUTPUT_FILE

5. Get the complete transcript using jq

jq '.output.text' $OUTPUT_FILE

Bonus: Use AI to summarize

This is an example of how we could use AI to extract key points. I am currently using Fabric by Daniel Miessler repository. In short it provides a set of convenience prompts to submit together with the input text, that can be accepted as a piped input, while not leaving the terminal.

One of my favorite is extract_wisdom which was designed for YouTube videos.

jq '.output.text' $OUTPUT_FILE | fabric -p extract_wisdom -s

As Google finally published API access Gemini 1.5 Pro on Vertex AI, I was very eager to get back to the task I've been struggling* since 2021 Christmas. Where I set out to extract messages from un-ideal screenshots of iMessage, Tinder, Instagram etc.

\Because it's my side hobby project so I haven't spent actual years working on it. But to be considerate, I did put a lot of time since then testing various ideas.*

While I mainly focused on Gemini, I also run parallel prompts to GPT 4 Turbo and Claude 3 Opus models to evaluate. The observations apply to them all.

The problem

...is that the screenshots are very random. They can contain many not wanted elements (a name of a mobile provider, UI elements), may not be perfectly cropped, and the design may have changed over time, including or excluding extra details.

This is exactly where powerful LLMs such as Gemini 1.5 Pro bring a big promise and thus big expectations on processing general unstructured information.

Goal

To successfully complete this job, the output, at a bare minimum, needs to contain:

• the message body including emojis,

• the correct sender (either "us" or the opponent),

• ignore any other text from UI elements - only from the actual messages of the conversation.

My observation notes

Saturday, April 13. I spent the last four hours integrating and testing Gemini 1.5 Pro to run OCR on screenshots. The approach was to have a detailed prompt on how to extract messages and run one request per image. So no matter how many screenshots (that is, how many tokens) a conversation may contain, it should work one by one.

I spent a lot of time testing, attempting various angles, and feeling really frustrated because it was not successful, and as Titanic once did, the time is gone.

Alignment

My first approach was to tell it whether the message is aligned to the left or to the right (padded on the right or on the left, respectively). This is how we intuitively tell who sent it. Even if I use black & white filter on my phone, which I love.

One thing that was not immediately obvious, but with debugging I found out, is that I had to explicitly tell it to judge from the viewer's perspective. Otherwise it would flip/mirror the sides to often. Even with temperature set to 0.

This proved to work rather great-ish. 7.5 out of 10 times.

The model is consistently inconsistent. It fails to achieve any consistent results. This is especially common when messages vary in length but are still short enough to fit a single line in the UI. It tends to shift the flow, whether it thinks a message appears on the right or left, resulting in poor parsing quality.

OpenAI has mentioned that their vision model is struggling with spatial reasoning. It seems to me that no matter the provider - they all are struggling as a team.

Color

In the context of an iMessage screenshot, I thought of asking about the color of the message bubble, since the opponent will always appear in gray. However when it fails to detect the alignment, it will also hallucinate the color - will reply it is gray, even though the message appears on the right with a blue bubble.

I also noticed that in some requests, it will output colors that don't exist or are turned into black and white. For example, with Google Gemini, sometimes it will say the message appears in a white bubble on a green app background, which never happened in any image. Super strange.

For the rest 4/10 times, it would think everything is in gray. So either the model is hallucinating, or Google is performing some kind of unclear pre-processing.

Furthermore, once in a while I observed the model spitting out the complete bullshit by mentioning the colors that are not seen at all; or making it seem as the image was further preprocessed and turned mono-chrome. If any of the providers are hiding this, it is a very unprofessional move to fail to communicate that.

The problem with background color is that emojis sent in certain cases are shown as tiny images with no background color at all. A hard rule like that would not work.

Emojis

Let's talk about 'em, the emojis. The model is very good for parsing text but only 50-50 for extracting and parsing emojis. They're often not the ones that actually appear and often will be missed out too.

To my surprise - and perhaps to yours too - no company has ever worked on extracting emojis except for The Hive AI. They have the best models I've seen hands-on. However they struggle with "I"s. These get missed out a lot or become a vertical symbol instead. Crazy.

Mirroring issue

Another one I could describe is the mirroring issue. It tends to flip the image in the middle of processing the image.

If a message from the opponent is short leading to a long message from "us", it will assign "our" message to the opponent and will continue putting the wrong sender for the rest of the parsing.

Were it to do that consistently, then all I need to do is flip it over. But no. It's in the middle haha.

Google AI Studio Playground

Furthermore, using Google AI Studio, I discovered that any subsequent requests in the chat that provided an image in the first request are not actually processing the image again.

I found this while attempting to debug and ask the model why it thinks a message is not blue and not aligned to the right when it is. It would spit out an answer based on the image description and assumptions about how messages would typically appear. It's quite obvious that all it gets is a description of once processed image.

This is rather frustrating, and I would blame Google for this. I've never seen this communicated, and if it is, it's definitely not clearly communicated.

I do read through documentation, and I'd like to think I only miss out on a minority of details. In other words, I tend to catch majority of details.

On the other hand, I was cheating too - using VPN to access otherwise geo-blocked playground (hence me waiting for Vertex AI). I guess that means we're even: 1 - 1.

Not ready for now

For now I conclude that although it is promising and exciting to parse a lot of information from screenshots that are otherwise not text-searchable, it is not implementable without human in the loop with an interface for review. At which, it would be marginally faster than a human alone.

To close with the opening,

powerful LLMs bring a big promise and set high expectations on processing general unstructured information but they fail to execute it consistently.

On the bright side, this is how I build my knowledge. Much more fun than reading books.

That being said, Elon Musk introduced his Grok 1.5 introducing vision capabilities. I would bet that this model should be good if he wants to use for self-driving Teslas.

Tips for working with LLMs

A few things that I found very useful and you should know too.

• Debug prompts by asking the model explain what it sees. Be it a text or be it an image. We see it differently so make no assumptions.

• Run parallel queries to all providers and make sure hyper parameters are the same. This way you can get ideas.

• Ask the model to tell the variables you want it to apply the logic for. Such as, the message is sent by us if it's in a blue bubble. So ask what the color the bubble is in.

• Save prompt versions and take breaks. It's easy to get lost a rabbit hole and deviate from the main approach. Taking a break can give you fresh eyes and fresh ideas. A prescription of 15 minute walk can do wonders.

That is it, my dear reader.
If you have any ideas, I would love them to hear.