2024 Technical Kick-off
TL;DR
We have identified a number of workstreams needed to ensure the sustainability of the technical assets embodied in the Product-Live platform. These include
- streamlining our applications, and eliminating certain redundant aspects
- Preparing the platform for the challenges of the years to come, and in particular for the increased volumes handled
- Rework some of our projects to make them easier to maintain in the future.
We have identified two large topic (regarding the effort needed to address them) and a number of medium and small topics. The large topics will require to be addressed in a sequential manner, while the medium and small topics can be addressed in parallel to some extent. The selected topics are described below.
Introduction
The aim of this exchange was to present the various key technical topics we had identified for 2024, and to select those we would like to prioritize this year.
Topics covered
The presentation used as a basis for our discussion can be accessed here: Technical Roadmap 2024.pptx
Subject 1: Rethinking file management within the platform
In a nutshell
Rethink and harmonize the way we manage files within the Product-Live platform (app.product-live.com, settings.product-live.com, Data Factory).
Expected benefits: simplification of the file management architecture and thus of the platform's maintenance, better management of the life cycle of these files (and thus a significant reduction in the volume of data stored), better control of the accessibility of these files, and better tracking of their usage (volume exchanged, number of downloads, etc.)
Github issue: 📚 [2024 Technical Roadmap] - Rethinking file management within the platform
Today
- There are 3 types of files within the platform (associated with items, job definitions, files generated by Data Factory).
- There's no easy way to distinguish them
- The life cycle of these files is not managed (e.g. deleting an item does not delete the associated files).
- It's not possible to limit the accessibility of these files, or to share them.
- Also, very few usage metrics are available (volume exchanged, number of downloads, etc.).
Objectives
- Define a new, more coherent file management architecture
- Manage file lifecycles (in particular, the many temporary files generated by Data Factory)
- Track usage of each account within the platform
Subject 2: Account creation tool
In a nutshell
Propose a tool enabling the company's various teams to create and edit different types of resources (accounts, users, tables, etc.) in bulk.
Expected benefits: Simplification of the account creation process, and thus a significant reduction in the time required to set up an account, and a reduction in the number of errors made during this process.
Github issue: 📚 [2024 Technical Roadmap] - Account Creation Tool
Today
- Numerous manipulations are required to completely setup an account.
- Within our development cycle, these manipulations are repeated many times (ticket qualification, pre-QA, QA on dev, QA on stage).
- Other teams also have to set up accounts (support teams, service teams, pre-sales teams, etc.) and face the same problems.
Objectives
- Propose a solution to facilitate account creation
- Define a standard format between teams for formalizing accounts
Subject 3: Eliminate direct CSS in our various applications
In a nutshell
Eliminate custom CSS in favor of more generic tools like Tailwind
Expected benefits: Better management of the application's style and a reduction in the complexity induced by too many CSS classes and thus a reduction in the time required to maintain the application and develop new features.
Github issue: 📚 [2024 Technical Roadmap] - Eliminate direct CSS in our various applications
Today
- Numerous css classes / files exist and we do not know easily when we browse code if they are still used and their impact.
Objectives
- Propose an alternative to this style management (tailwind, stylex, etc)
- Keep refactoring front monorepo (slices, functional components etc)
Subject 4: Refactor React Class Component
In a nutshell
Refactor the class components of our React applications into functional components in order to take advantage of the latest features of React.
Expected benefits: Alignment with the best practices recommended by React, which will enable us to benefit from a more modern tool, to test our components more easily, and also to make all our projects more maintainable.
Github issue: 📚 [2024 Technical Roadmap] - Refactor React Class Component
Current Situation
- Our codebase use lot's of Legacy Class component
- Rely on VanillaJS queries: some component use vanillaJS for DOM interactions making them hard to reuse
- Limit what we can do and force Redux usage: because hooks (custom and lib) cannot be employed within class components.
- Oversized: Not enough decomposed in smaller components
- Not feature-centric: They are not organized in a feature-centric directory structure
- Lack of Isolation: They are often tightly coupled, making isolated testing and reuse challenging
Objectives
- Transition to Functional Components for more concise and modern code
- Make smaller Isolated components (more testable, more re-usable, more readable)
- Use Separation of Concern
- Make Stories (storybook) to consolidate testing strategy
- Prepare for more modern technologies like Zustand to replace Redux or React Query
Subject 5: Calcifer & graphql API
In a nutshell
Offer a GraphQL API to ease the integration of Calcifer into our various applications & the use of our API inside the Data Factory platform.
Expected benefits: Simplification of the integration of Calcifer into our various applications, and make the use of our APIs inside the Data Factory platform easier.
Github issue: 📚 [2024 Technical Roadmap] - Calcifer & graphql API
Today
- More and more calcifer endpoints with specific output / input when we might want different data for an endpoint depending on the context
Objectives
- Investigate the pro / cons of a graphql solution to this problem
Subject 6: Generalize the use of Item instead of row data in business code
In a nutshell
Today we use different formats to represent a tim (or a line within the grid), the aim being to rationalize this and use a single format.
Expected benefits: Simplification of the code, and better maintainability
Today
- Raw data is used in most business code (publication, mapping, formulas …), when the hydrated Item would be easier to work with. It is also the entity served by calcifer
Objectives
- Have one format for item manipulation
- Simpler code
- Secondary database friendly
- Allow removal of Haku
Subject 7: Create a read DB with pre-rendered items
In a nutshell
Create a read-only database with pre-rendered items to improve the performance of the platform and its scalability & maintainability.
Expected benefits: Better performance when reading items, prepare the platform to handle larger volumes of data, and allow to propose a more complete find / search feature.
Github issue: 📚 [2024 Technical Roadmap] - Create a read DB with pre-rendered items
Today
- Items / raw data are hydrated at every request, preventing complex generic queries, and slow down reading item
Objectives
- Adding a read DB
- Better perfs when reading
- Allows simpler business code logic because they are ran on richer objects
- Event sourcing friendly if we ever chose to go this way
Subject 8: Unify Formula / Mapping / CF
In a nutshell
In the past, we have implemented three distinct functions (Formula, Conditionnal formatting and mapping), which could be replaced by a single feature, in order to simplify maintenance and development.
Expected benefits: Remove two features to only keep one, and thus simplify the maintenance and development of the platform.
Today
- The broad feature of field automation is covered by 3 features formula, mapping and CF. This make dev, tests, and QA harder.
Objectives
- Have 1 feature
field automationwith different flavors
Subject 9: Remove Haku & Kanoashi endpoints, replace them by Calcifer
In a nutshell
Remove Haku and Kanoashi in profit of Calcifer (or at least use the same entity and DTOs for all of them)
Expected benefits: Simplification of the platform's architecture, and better maintainability
Github issue: 📚 [2024 Technical Roadmap] - Converge our HAku, Kaonashi and Calcifer APIs
Today
- The haku API was made for our own use, and suffers from a patchwork effect making it harder to maintain and unusable outside out team
- Similar but better endpoints are being added to calcifer
Objectives
- Remove haku in profit of calcifer
Subject 10: Switch to the Orkes-managed version of Conductor
In a nutshell
At the end of 2023, Orkes took over the lead in the development of the Conductor tool and offers a managed version. The aim of this project is to delegate the management of this tool to Orkes.
Expected benefits: Eliminate the costs associated with maintenance and run of Conductor, take advantage of developments made by Orkes (and available only on their managed version) and decommission a number of services on our side, such as periodic job management.
Today
- We build and operate our own conductor
- We do not carry out any development work directly within conductor
- We carry out 1 to 2 conductor updates a year
Objectives
- Take advantage of orkes' ongoing investment in the conductor platform
- Save yourself the costs induced by the maintenance of conductor
- Eventually decommission some of our services whose scope is covered by the orkes version of conductor (e.g. periodic job scheduling with ged).
Subject 11: Switch all mailings to the Knock platform
In a nutshell
In 2023 we deployed the Knock tool to manage notifications within the platform. This solution triggers the sending of transactional e-mails and duplicates a number of other tools. The aim is to harmonize the sending of transactional e-mails so as to keep just one single tool.
Expected benefits: Harmonization of our transactional emails, elimination of redundant tools, and empowering the product team to continuously improve the content of the emails we send to our customers.
Github issue: 📚 [2024 Technical Roadmap] - Switch all mailings to the Knock platform
Today
- Mails can be sent by different services: Tombo / Keycloak / Knock
- Mails are not localized (for the most part)
- Development is required for all template modifications
Objectives
- Standardize transactional e-mail management
- Discontinue Tombo
- Giving the product team control over mail templates
- Standardize our transactional e-mails
Phase 1 - Selection of the topics to be prioritized
Approach: each member of the team was able to express their opinion on the topics presented, and to vote for the topics they considered to be the most important. Each person had 5 votes to distribute as they saw fit.
Results
The subejct #1 Rethinking file management within the platform was not voted on by the team, as it was identified as a priority topic for the sustainability of the platform.
The most voted topics were:
The following topics did not receive enough votes to be considered as a priority, and will not be addressed in 2024:
- #6 Generalize the use of Item instead of row data in business code
- #10 Switch to the Orkes-managed version of Conductor
Phase 2 - Rough estimate of the workload associated with these topics
Large subjects
- The most consequential subjects (or at least the subjects with the most uncertainty) are:
- It is estimated that these two subjects should occupy one to two people, over a period of several weeks, spread over 2 quarters at most. They should not be processed, or at least delivered, at the same time, given the heavy workload they will require in terms of quality assurance and development.
Medium subjects
- We estimate that the following topics should keep one or two people busy for several weeks, spread over a quarter at the most:
Small subjects
- The following topics may be handles by one or two persons, in a matter of days, and may be processed in parallel:
Phase 3 - Define an approximate calendar for the year
- The two large subjects should be addressed in a sequential manner, with the first one starting by the end of Q1 and the second one starting in Q3
- The medium subject #4 Refactor React Class Component can be dealt with throughout the year, whenever we need to work on a component that needs to be refactored
- Some time has already been spent on topic #2 Account creation tool, so to build on this initial momentum, this topic should be dealt with at the start of the new year
- The small topics can be addressed in parallel, and should be dealt with as soon as possible
- Each topic has been assigned a leader (indicated by a star ⭐), who will be responsible for coordinating efforts on that topic
Conclusion
The topics selected for 2024 are as follows:
- Two main, highly structuring topics that will require a major effort:
- And a number of medium and small topics that may be addressed in parallel: