Generic implementation for executing expressions of a kind: 1 + 2 * 4 + 3

The image has nothing to with the content

Goals

We will implement an algorithm to convert an infix expression into a postfix expression, which we later will execute to produce the result

Why Do We Need This?

One of the applications is to execute human-readable expressions. Consider implementing a basic command-line calculator.


TL;DR: see an example here: https://github.com/anatolyD/distributed-jacoco

All-in-one on the left, awesome distributed on the right! 🍿🍿🍿

Your mvn clean verify takes forever to run? Your teammates eager to spend extra Github Action(GHA) minutes in favor of let my PR checks run faster!? Say no more. Go and re-write your tests to take less time. But before you do that there is another way.

Here is what we will do:

  1. Split test runs across multiple GHA runners
  2. Merge overall reports from all of them into a single place

Splitting Tests

Let’s base it on the fact that we wrote all the code (and hence tests) in one huge module. …


Solving a challenge of inferring schema for 3GB file without reading it

Photo by Vitalii Chernopyskyi on Unsplash

Problem

To infer schema in AWS we can set a special option that works pretty well for small to medium file sizes:

.option("inferSchema", "true")

It takes one extra parsing run through the whole file before actually processing it. You do not really want that given slow connection and when you really want to just peek on the output

Solution

We will cut off the first 1000 records and will hope that this is enough to give a hint to a recipient about the schema. …


FailFast, Permissive, and Drop Malformed all behave differently

Hard choice!

Input

For the use cases here and below we will use this CSV file:

We can spot that for the two header columns row 4 and 6 have extra separators thus will break our parsing 🚫

Spark will be told to use schema age STRING, listen_mom STRING which definitely should cause some troubles, let’s see how different modes parse it.

FAILFAST mode

In this mode we expect an exception to be thrown on the first malformed record

Yes, Spark and Kotlin — friends forever

And there we go, Spark recommends us to use PERMISSIVE…


The training set every git-user must know. Beginner users, of course, true masters don’t need version control systems. 🥋

No animals were harmed in the making of this article

What we will cover is:

  1. Creating a branch for changes
  2. Pushing completed changes
  3. Rewriting history
  4. Resolving merge conflicts
  5. Cherry-picking

So, you got a task to do and you want to follow git-flow, GitHub-flow, or trunk-based-development, say something that doesn’t let you commit straight to the master branch.

1. Create your own branch for task feature/XXXX_description:

When you open a new branch

2. Push your changes to the remote server

Let’s do some changes to the codebase.

echo "New ideas come with painful struggle" >> readme.md
echo "Many new stuff" > new-file.md
ideal scenario without side untracked files. Lovely

Now you can open…

Anatoly Danilov

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store