Whenever I have to deal with anything “data science” or “data engineering” or what the fuck we call it this week, maybe it is all (sarcastic Homer Simpson voice) AI, the whole notebook stuff drives me nuts.

Like I think they are fun for some stuff, but it is so weird for a lot of use cases at work. Mostly because we have all of these great developer tools over here, and then most of them aren’t really usable within the notebooks.

Is this a problem because I’m more of a “software engineer” (I refuse to use the engineer title myself) than a “data engineer”, yes. But here is the reality of doing software in any company everything has to talk to each other in some way.

Development, operations, “data” all has to do work on the others work sometimes. Or a lot of the time. Or you have to take on the role of some of the others for a while.

When I see all of the code blocks in the notebooks I just think: can I have these in a git repo as files that are linked into the main document when you work on it as a notebook? So that I can write the code in PyCharm or VS Code or whatever?

I like literate programming by the way. But I think there is a place for it and a place where it isn’t a good fit. But since everything is based on it at this point, it should be destructed into a way where it is less of a pain for different disciplines to deal with.

I guess most people have been in this situation. You do something that bothers someone else, and you either wouldn’t get bothered by it yourself because not a big deal or because you don’t understand how it is offensive / how it was understood that way.

Here is the thing.

When someone else reacts in a negative way to what you do, it doesn’t really matter if you agree with them or “get it”, just apologize and try to understand it well enough so that you can try to avoid it in the future.

Everyone is different. Some of us have experienced a lot of things that results in triggers, different reactions or more extreme reactions to things.

Accept the boundaries people try to maintain, and try to not think about everything in relation to your perspective, but rather that everyone are different and one gets a lot further by not framing everything different from you as “weird”.

The way I use git with feature branches, rebasing, interactive rebase etc I’ve found it incredible useful to do my commits in multiple phases.

When I work on something I commit all the time, usually with messages that’s only useful to me.

Then when I’m ready to send it to review I will re do them, either as in squashing or something as in resetting all of them and re-doing them. Typically having separate commits for lining, fixing kind of related issues, making the project ready for the change I’m doing, the actual change, kind of related fixes after implementing it, unrelated fixes etc.

Useful both for keeping track of what I’m doing while working, and then making sure the git history makes sense for other people after.

Sometimes I come across things that I should have found way earlier because of how awesome it is.

One of them is jsonl. It is basically a text file where every line is an independent chunk of json data.

It is extremely useful for a bunch of reasons, like when you want to append things to a log without having to read, deserialize, update, serialize and write. Or if you want to parallelize to processing of data etc.

The thing I hate-enjoy the most about making Apple Shortcuts is how it encourages (because it lacks real complex logic or data structures) to write logic that takes shortcuts and kind of avoids checking all cases. Like with what I was doing when I got this post idea:

• Two variables (a and b) that can have three states: Green, Yellow, Red.
• Where the global state should be Red if either is Red, or Yellow if either is Yellow otherwise Green
• Easy with a or something more fancy like a data structure that holds the results for each possible state
• With Shortcuts I always find myself doing something like this

1. Set Status to A
2. Then a shit load of nested if
3. If Status != Red
4. If B != A
5. If B != Green

In the same way you would in places where performance is very important and you only do what you have to do.

I could have re-written it to be numbers and found the largest ones, and then converted that to the statuses. But that isn’t really the point here: can we have some real if logic soon?!

I have had the “pleasure” of experiencing a lot of these solutions over the years. And I guess an important thing to keep in mind is that these tools probably isn’t that fair to compare with the kind of development I typically do. Either because of staffing, money or other reasons.

But from a user side of things I always find them incredibly slow with a user experience that leaves A LOT to be desired. While on the development side it is also slow to set them up, with surprising little of the easy duplication and re-use you see in conventional software development and also the apps are very slow and cumbersome in themselves.

Again, I’m most likely not the target audience.

It is really important to understand how you work when you try to manage yourself, set goals, improve things etc.

Some people are need to pretend like everything is fun, I just focus more on why I am doing it.

I keep track of all the data and have lists from milestones and all of that to drive most of what I do. Hide most of it, and just have what I am working on now in front of me. And typically don’t care too much of what I have done and just have moved on to new goals as I’m reaching the old ones.

Other people want to celebrate things more.

It is kind of like the different kind of people at the gym. I just go alone and do it every day because I want to be stronger, fitter etc. Others have a friend they go with, use personal trainers or go to a class etc. It all depends on your drive.

To understand how to manipulate (or “motivate” if you do a corporate training thingy) yourself is essential to move forward

Many apps have some onboarding process to make it easier for newer users to get started.

Please.

Let me have a way to avoid having to run through it every time I install your app on a new device.

Either by properly detecting when users have gone through it before and letting me log-in before completing it. And ideally let me skip it and re-visit it whenever I want to.

If you make an app, and provide some sample data as the onboarding process, give me the option to keep it or not, so I don’t have to spend a bunch of time deleting stuff.