Generic implementation for executing expressions of a kind: 1 + 2 * 4 + 3
We will implement an algorithm to convert an infix expression into a postfix expression, which we later will execute to produce the result
One of the applications is to execute human-readable expressions. Consider implementing a basic command-line calculator.
TL;DR: see an example here: https://github.com/anatolyD/distributed-jacoco
mvn clean verify takes forever to run? Your teammates eager to spend extra Github Action(GHA) minutes in favor of
let my PR checks run faster!? Say no more. Go and re-write your tests to take less time. But before you do that there is another way.
Here is what we will do:
Let’s base it on the fact that we wrote all the code (and hence tests) in one huge module. …
Solving a challenge of inferring schema for 3GB file without reading it
To infer schema in AWS we can set a special option that works pretty well for small to medium file sizes:
It takes one extra parsing run through the whole file before actually processing it. You do not really want that given slow connection and when you really want to just peek on the output
We will cut off the first 1000 records and will hope that this is enough to give a hint to a recipient about the schema. …
FailFast, Permissive, and Drop Malformed all behave differently
For the use cases here and below we will use this CSV file:
We can spot that for the two header columns row 4 and 6 have extra separators thus will break our parsing 🚫
Spark will be told to use schema
age STRING, listen_mom STRING which definitely should cause some troubles, let’s see how different modes parse it.
In this mode we expect an exception to be thrown on the first malformed record
And there we go, Spark recommends us to use PERMISSIVE…
The training set every git-user must know. Beginner users, of course, true masters don’t need version control systems. 🥋
Let’s do some changes to the codebase.
echo "New ideas come with painful struggle" >> readme.md
echo "Many new stuff" > new-file.md
Now you can open…