Kiba author here! This is an important question, thanks for asking it!
MY QUESTION : Should I use Rails on that ETL project?
By default, I would recommend to start with a separate project (like a kind of "macro-service" approach), unless you have important things (more than just RSpec & ENV setup) to reuse from the Rails app.
If there is an important expected coupling between the app and the ETL (e.g. by "scheduled jobs" you mean jobs triggered through Sidekiq, to react to events, or you have classes shared between the 2 projects), then you can place the ETL in a etl subfolder of your Rails app, for instance, to provide a bit of separation and leave the opportunity to split the code out later if it becomes a better path (this is a middle ground I'm using on some projects).
If it is not the case, though, and the data pipeline is expected to become large and live its own life, you can instead split it to its own project.
Using it means we can apply the same folder structure as the other repos, use RSpec all the same etc.
You can use RSpec or minitest from a dedicated ETL (pure Ruby) project too, introduce a notion of ETL_ENV (development, test, production), build your own ENV-based (or file based) configuration with dotenv or similar, and support cron jobs from there too if you need that.
Pure Ruby projects can be structured just like a Rails app, and there is usually less magic (more explicit), which is helpful.
It also appeared to me that Rails changes the way classes like Hash act.
I would actually recommend to use an "explicit" approach about depending about that. Today I prefer to "cherry-pick" the exact extensions I need, at the top of each file (as described here).
One last word, you can test out Kiba ETL pipelines just as much as your individual ETL components, and I would recommend to do so (I will cover that in a future blog post), since it helps moving things around and upgrading Ruby with ease, and generally scale the team of developers easily (CI + tests).
I hope this provides enough guidance for you to take a decision on this, if this is not the case, please comment out!
