Integration refers to the process of combining software parts (or subsystems) into one system. An integration framework is a lightweight utility that provides libraries and standardized methods to coordinate messaging among different technologies. As software connects the world in increasingly more complex ways, integration makes it all possible facilitating app-to-app communication. Learn more about this necessity for modern software development by keeping a pulse on the industry topics such as integrated development environments, API best practices, service-oriented architecture, enterprise service buses, communication architectures, integration testing, and more.
When It’s Time to Give REST a Rest
Build a Time-Tracking App With ClickUp API Integration Using Openkoda
Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices. In the dynamic and rapidly evolving landscape of cloud-native and SaaS-driven software development, the process of API generation plays a pivotal role in accelerating time to market (TTM) and competitive advantage by rapidly creating APIs. This crucial function now converges with low- and no-code platforms, ushering in new ways of streamlined development processes. At the forefront of this evolution stands generative AI (GenAI), which offers unprecedented speed and flexibility in API creation. In this article, we embark on a journey to explore the transformative potential of generative AI and low- and no-code platforms in API generation by highlighting their roles in fostering innovation and expediting time to market. Additionally, we delve into strategies for effective implementation, compliance considerations, enterprise patterns, and the future trajectory of no-code API generation. Refer to Figure 1 for an illustration of the key components and architecture involved in this synergy: Generative AI – These algorithms autonomously generate code snippets by analyzing large datasets. Low-code/no-code platform – This encompasses the visual interface and pre-built components enabling API design without manual coding. API generation components – These leverage GenAI capabilities to automate API code generation based on user-defined specifications and prompts. API output – The final generated API code that is ready for deployment and integration into software applications. Generative vs. Conventional APIs in the Modern Software Landscape The debate between generative and conventional APIs has intensified, particularly concerning their impact on time to market and skill requirements. Generative APIs — powered by advanced technologies like artificial intelligence (AI) on top of low- and no-code platforms — have gained traction for their ability to automate the API generation process, promising faster TTM and reducing the skill threshold required for development. Conversely, conventional APIs — built through manual coding processes — typically demand a higher level of technical expertise and entail longer development cycles. In contrast, conventional APIs offer flexibility, tailored customization, and, hence, better capabilities for handling complexity. Table 1. Generative vs. conventional APIs Aspect Generative APIs Conventional APIs Time to market Rapid development cycles due to automation Lengthy development timelines Skill requirements Lower skill threshold; accessible to non-coders Require proficient coding skills Handling complex logic May struggle with replicating complex logic accurately Excel in handling complex logic and intricate business rules Integration complexity Simplified integration with existing systems May be challenging and time consuming Maintenance effort Reduced due to automation Require ongoing manual updates and maintenance Innovation potential Enable rapid experimentation and innovation May be limited due to manual coding and testing processes Error handling Automated but may be less customizable Fully customizable Legacy system integration May require additional effort Better compatibility and integration capabilities Creative freedom Limited due to automated processes Flexibility for developers Table 1 highlights the trade-offs between the two approaches, emphasizing the potential to revolutionize API development by expediting TTM and democratizing access to software development. Moreover, by lowering the skill threshold required for API development, these tools empower a broader range of individuals to contribute to the creation of APIs while developers can focus on higher-level tasks. The Synergy of Generative AI and Low and No Code for API Generation The integration of GenAI with low and no code for API generation signifies a significant advancement in API development/generation practices, particularly in terms of efficiency and accessibility. GenAI employs sophisticated large language model (LLM) algorithms that learn to autonomously analyze large datasets on enterprise (an organization's internal context) and open source patterns. When combined with low- and no-code platforms, GenAI enhances these tools by providing developers AI-generated code components that can be easily incorporated into their applications. This integration streamlines the development process, allowing developers to leverage pre-built AI-generated modules to accelerate prototyping and customization. Overall, the integration of GenAI with low- and no-code platforms revolutionizes API development, empowering organizations to innovate rapidly and deliver high-quality solutions to market with unprecedented speed and efficiency. Additionally, this amalgamation profoundly impacts the daily operations and responsibilities of engineers and developers, empowering them to innovate, streamline workflows, and adapt to the evolving demands of modern development practices. Figure 1. Architectural overview of GenAI with low- and no-code platforms for automated API generation Charting the Future: The Trajectory of No-Code API Generation As businesses prioritize agility and efficiency, the rising adoption of automated no-code API generation is set to transform development processes, therefore streamlining workflows and expediting TTM. With no-code platforms now able to accurately generate complex APIs by leveraging GenAI, this trajectory foresees a transition to more intuitive and potent development tools, which, in turn, will empower organizations to innovate swiftly and deliver top-notch solutions with unparalleled speed and efficiency. Democratizing Development As organizations adopt these innovative solutions, the democratization of development gains momentum. No-code API generation platforms empower individuals with diverse technical backgrounds to engage in API creation, promoting collaboration and inclusivity. By lowering entry barriers and reducing reliance on traditional coding skills, these platforms foster a future where API development is accessible and collaborative. This democratization accelerates innovation and ensures that a broader range of voices and perspectives contribute to the development process, resulting in more inclusive and impactful solutions. Compliance and Security In the domain of automated no-code API generation, organizations must prioritize compliance and security to uphold the integrity and reliability of their APIs. Compliance entails adhering to regulatory requirements such as GDPR, HIPAA, and PCI-DSS, as well as industry-specific benchmarks like ISO 27001 or SOC 2. Additionally, robust security measures — including authentication and encryption protocols — are essential for safeguarding against unauthorized access, secret rotations, and cyber threats. Prioritizing compliance and security enables organizations to mitigate risks, protect data, and uphold stakeholder trust. Business Transformation No-code API generation empowers businesses to swiftly adapt to market changes by facilitating rapid API creation and deployment without extensive coding. These tools and platforms enable developers to iterate quickly on ideas, prototypes, and solutions, expediting the development cycle and enabling organizations to respond promptly to market demands. By automating manual coding tasks and streamlining development processes, these tools free up developers' time and resources, allowing them to focus on more strategic tasks such as innovation, problem solving, and optimization. Additionally, these platforms democratize the development process by facilitating collaboration among cross-functional teams with varying levels of technical expertise, fostering scalability and competitiveness. Embracing no-code API generation is essential for driving meaningful transformation and maintaining a competitive edge in today's dynamic digital landscape. Challenges of Automated API Generation While these tools offer streamlined development processes, they come with hurdles that must be addressed. The following chart highlights key technical challenges, providing insights for effective implementation. Table 2. Challenges of automated API generation Challenge Description Handling complex data structures and business logic Handling complex data structures and business logic may pose challenges in integrating diverse data, validating data, and handling versioning effectively. Integration dependencies Automated tools may struggle to integrate seamlessly with existing systems and external APIs due to data format disparities, API versioning conflicts, and limited customization options. Security vulnerabilities Automated processes of low- and no-code results may be prone to potential vulnerabilities such as inadequate access controls, insecure configurations, and absence of data encryption, as well as vulnerabilities within third-party integrations. Limited flexibility Limited flexibility arises from predefined templates or patterns, constraining customization and adaptability to specific project requirements, potentially impacting functionality and scalability. Future proofing Low- and no-code API platforms are abstract in nature and can result in vendor lock-in; adaptability to evolving technologies as well as ensuring long-term support and compatibility can be challenging. Strategies and Guidelines for Generative No-Code API Development Strategies and guidelines provide a roadmap for leveraging AI-driven tools and low- and no-code platforms effectively. These strategies and guidelines encompass comprehensive planning, iterative development, and collaborative approaches, thus ensuring streamlined workflows and accelerated TTM while prioritizing automation, scalability, and security. Table 3. Developing generative no-code APIs Key Aspects Strategies and Guidelines Comprehensive planning Plan thoroughly by defining clear objectives and requirements up front to ensure alignment with business goals and user needs. Iterative development Adopt an iterative development approach, allowing for continuous feedback, testing, and refinement throughout the development process. Collaborative development Foster collaboration between technical and non-technical stakeholders, encouraging cross-functional teams to contribute to API design and development. Embrace automation Leverage automation tools and features provided by no- and low-code platforms to streamline development tasks and increase productivity. Ensure scalability Design APIs with scalability in mind, anticipating future growth and ensuring that the architecture can support increased demand and usage over time. Prioritize security Implement robust security measures to protect APIs from potential threats, including data breaches, unauthorized access, and injection attacks. Testing and validation Implement rigorous testing and validation processes to ensure the reliability, functionality, and interoperability of APIs across different platforms and environments. Conclusion As we cast our gaze toward the future shaped by cloud-native and SaaS-driven development, the integration of generative AI with low- and no-code platforms emerges as a catalyst for innovation. This symbiotic relationship not only revolutionizes API generation but also bestows developers with unprecedented flexibility and efficiency. Embracing automation and innovation will be pivotal in meeting the evolving market demands and expediting TTM. This trend represents more than just a leap in technological prowess; it signals a paradigm shift in the ethos of API development, where the context of creativity and efficiency converge harmoniously. Ultimately, developers and engineers are empowered by automated API generation tools, enabling them to rapidly translate ideas into prototypes and solutions, thus expediting the development cycle. This capability positions engineering and development teams to respond promptly to market demands and feature requirements, fostering experimentation and innovation. By automating manual coding tasks and streamlining development processes, these tools unlock opportunities for organizations to gain a competitive edge by delivering value to customers with unparalleled speed. Despite inevitable challenges, such as compliance and security considerations, the trajectory of automated API generation remains on a path of progress. Embracing strategic guidelines and proactively addressing challenges, businesses can harness the transformative potential of automated API generation to shape the future of software development and technology trends. This is an excerpt from DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices.Read the Free Report
Imagine building a complex machine with numerous independent parts, each performing its function, but all needing to communicate effectively with each other to accomplish a task. This is the challenge we face when designing cloud-native applications, which consist of interconnected microservices and serverless components. In this article, we explore the specifics of designing robust and resilient communication systems that can effectively coordinate these independent elements both within and beyond the application boundaries. These finely-grained services engage in internal and external interactions, employing various communication methods, synchronous or asynchronous. In synchronous communication, a service invokes another service using HTTP or gRPC, awaiting a response within a specified timeframe before proceeding. Conversely, asynchronous communication involves exchanging messages without expecting an immediate response. Message brokers such as RabbitMQ or Kafka serve as intermediaries, buffering messages to ensure reliable delivery. In cloud-native applications, embracing a combination of communication patterns is often a practical approach. Let's begin with synchronous communications first. What Is Synchronous Communication? Synchronous communication is like a conversation. One service (let’s call it Service A) initiates a request and then waits for a response from another service (Service B) or external APIs. This is akin to asking a question and waiting for an answer. Service A sends a request over HTTP and waits. It’s either waiting for a response from Service B or for a maximum waiting time to expire. During this waiting period, Service A is temporarily blocked, much like a person who pauses their activities to wait for a response. This pattern, often referred to as a request-reply pattern, is relatively simple to implement. However, using it extensively can introduce challenges that require careful consideration. Synchronous communication in the cloud Challenges of Synchronous Communication While synchronous communication is a powerful tool in our cloud-native toolkit, it comes with its own set of challenges that require careful consideration. Temporal Coupling Excessive reliance on synchronous communication throughout the solution can lead to temporal coupling issues. It occurs when numerous synchronous calls are chained together, resulting in extended wait times for client applications to receive responses. Availability Dependency Synchronous communication necessitates simultaneous availability of all communicating services. If backend services are under unexpected loads, client applications may experience failures due to timeout errors, impacting overall performance. Network Quality Impact Network quality can directly impact the performance of synchronous communication, including available bandwidth and the duration required for responses to traverse between serving backend services. Despite these challenges, synchronous communication can prove invaluable in specific scenarios. Let us explore some use cases in the next section where synchronous communication might be the better choice. When To Use Synchronous Communication Here are some situations where using synchronous communication can prove to be a better choice. Real-Time Data Access or Guaranteed Outcome When immediate or real-time feedback is needed, synchronous communication offers efficiency. For instance, when a customer places an order on an e-commerce website, the e-commerce front end needs to check the inventory system to ensure the item is in stock. This is a synchronous operation because the application needs to wait for the inventory system’s response before it can proceed with the order. Orchestrating Sequence of Dependent Tasks In cases where a service must execute a sequence of tasks, each dependent on the previous one, synchronous communication maintains order. It is specifically appropriate for workflows where task order is critical. Maintaining Transactional Integrity When maintaining data consistency across multiple components is vital, synchronous communication can help maintain atomic transactions. It is relevant for scenarios like financial transactions where data integrity is paramount. Synchronous communication is a powerful tool and has its challenges. The good news is that we also have the option of asynchronous communication—a complementary style that can work alongside synchronous methods. Let us explore this further in the next section. What Is Asynchronous Communication? Asynchronous communication patterns offer a dynamic and efficient approach to inter-service communication. Unlike synchronous communication, asynchronous communication allows a service to initiate a request without awaiting an immediate response. In this model, responses may not be immediate or arrive asynchronously on a separate channel, such as a callback queue. This mode of communication relies on protocols like the Advanced Message Queuing Protocol (AMQP) and messaging middleware, including message brokers or event brokers. This messaging middleware acts as an intermediary with minimal business logic. It receives messages from the source or producer service and then channels them to the intended consuming service. Integrating message middleware can significantly boost the resilience and fault tolerance of this decoupled approach. Asynchronous communication encompasses various implementations. Let us explore those further. One-To-One Communication In a One-to-One message communication, a producer dispatches messages exclusively to a receiver using a messaging broker. Typically, the message broker relies on queues to ensure reliable communication and offer delivery guarantees, such as at least once. The implementation resembles a command pattern, where the delivered messages act as commands consumed by the subscriber service to trigger actions. Let us consider an example of an online retail shop to illustrate its use. An online business heavily depends on the reliability of its website. The pattern provides fault tolerance and message guarantees, ensuring once a customer has placed an order on the website, the backend fulfillment systems receive the order to be processed. The messages broker preserves the message even if the backend system is down and will deliver when these can be processed. For instance, in an e-commerce application, when a customer places an order, the order details can be sent as a message from the order service (producer) to the fulfillment service (consumer) using a message broker. This is an example of one-to-one communication. Asynchronous One-to-One communication in the cloud An extension of the one-to-one message pattern is the asynchronous request-reply pattern. In this scenario, the dispatcher sends a message without expecting a response. But in a few specific scenarios, the consumer must respond to the producing service, utilizing the queues in the same message broker infrastructure queues. The response from the consumer may contain additional metadata such as ID to correlate the initial request or address to respond. Since the producer does not expect an immediate response, an independent producer workflow manages these replies. The fulfillment service (consumer) responds back to the frontend order service (producer) once the order has been dispatched so that the customer can be updated on the website. Asynchronous One-to-One Request Reply communication in cloud The single consumer communication comes in handy when two services communicate point to point. However, there could be scenarios when a publisher must send a particular event to multiple subscribers, which leads us to the following pattern. One-To-Many Communication This communication style is valuable when a single component (publisher) needs to broadcast an event to multiple components and services (subscribers). The one-to-many communication uses a concept of topic, which is analogous to an online forum. It is like an online forum where multiple users can post articles, and their followers can read them in their own time, responding as they fit. Similarly, applications can have topics where producer services write to these topics, and consuming services can read from the topic. It is one of the most popular patterns in real-world applications. Consider again the e-commerce platform has a service that updates product prices and multiple services need this information (like a subscription service, a recommendation service, etc.), the price update can be sent as a message to a topic in a message broker. All interested services (subscribers) can listen to this topic and receive the price update. This is an example of one-to-many communication. Several tools are available to implement this pattern, with Apache Kafka, Redis Pub/Sub, Amazon SNS, and Azure Event Grid ranking among the most popular choices. Asynchronous One-to-Many communication in cloud Challenges of Asynchronous Communication While asynchronous communication offers many benefits, it also introduces its own set of challenges. Resiliency and Fault Tolerance With numerous microservices and serverless components, each having multiple instances, failures are inevitable. Instances can crash, become overwhelmed, or experience transient faults. Moreover, the sender does not wait for the message to be processed, so it might not be immediately aware if an error occurs. We must adopt strategies like: Retry mechanisms: Retrying failed network calls for transient faults Circuit Breaker pattern: Preventing repeated calls to failing services to avoid resource bottlenecks Distributed Tracing Asynchronous communication can span multiple services, making it challenging to monitor overall system performance. Implementing distributed tracing helps tie logs and metrics together to understand transaction flow. Complex Debugging and Monitoring Asynchronous communications can be more difficult to debug and monitor because operations do not follow a linear flow. Specialized tools and techniques are often required to effectively debug and monitor these systems. Resource Management Asynchronous systems often involve long-lived connections and background processing, which can lead to resource management challenges. Care must be taken to manage resources effectively to prevent memory leaks or excessive CPU usage. Understanding these challenges can help design more robust and resilient asynchronous communication systems in their cloud-native applications. Final Words The choice between synchronous and asynchronous communication patterns is not binary but rather a strategic decision based on the specific requirements of the application. Synchronous communication is easy to implement and provides immediate feedback, making it suitable for real-time data access, orchestrating dependent tasks, and maintaining transactional integrity. However, it comes with challenges such as temporal coupling, availability dependency, and network quality impact. On the other hand, asynchronous communication allows a service to initiate a request without waiting for an immediate response, enhancing the system’s responsiveness and scalability. It offers flexibility, making it ideal for scenarios where immediate feedback is not necessary. However, it introduces complexities in resiliency, fault tolerance, distributed tracing, debugging, monitoring, and resource management. In conclusion, designing robust and resilient communication systems for cloud-native applications requires a deep understanding of both synchronous and asynchronous communication patterns. By carefully considering the strengths and weaknesses of each pattern and aligning them with the requirements, architects can design systems that effectively coordinate independent elements, both within and beyond the application boundaries, to deliver high-performing, scalable, and reliable cloud-native applications.
Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices. Microservices-based applications are distributed in nature, and each service can run on a different machine or in a container. However, splitting business logic into smaller units and deploying them in a distributed manner is just the first step. We then must understand the best way to make them communicate with each other. Microservices Communication Challenges Communication between microservices should be robust and efficient. When several small microservices are interacting to complete a single business scenario, it can be a challenge. Here are some of the main challenges arising from microservice-to-microservice communication. Resiliency There may be multiple instances of microservices, and an instance may fail due to several reasons — for example, it may crash or be overwhelmed with too many requests and thus unable to process requests. There are two design patterns that make communication between microservices more resilient: retry and circuit breakers. Retry In a microservices architecture, transient failures are unavoidable due to communication between multiple services within the application, especially on a cloud platform. These failures could occur due to various scenarios such as a momentary connection loss, response time-out, service unavailability, slow network connections, etc. (Shrivastava, Shrivastav 2022). Normally, these errors resolve by themselves by retrying the request either immediately or after a delay, depending on the type of error that occurred. The retry is carried out for a preconfigured number of times until it times out. However, a point of note is that the logical consistency of the operation must be maintained during the request to obtain repeatable responses and avoid potential side effects outside of our expectations. Circuit Breaker In a microservices architecture, as discussed in the previous section, failures can occur due to several reasons and are typically self-resolving. However, this may not always be the case since a situation of varying severity may arise where the errors take longer than estimated to be resolved or may not be resolved at all. The circuit breaker pattern, as the name implies, causes a break in a function operation when the errors reach a certain threshold. Usually, this break also triggers an alert that can be monitored. As opposed to the retry pattern, a circuit breaker prevents an operation that’s likely to result in failure from being performed. This prevents congestion due to failed requests and the escalation of failures downstream. The operation can be continued with the persisting error enabling the efficient use of computing resources. The error does not stall the completion of other operations that are using the same resource, which is inherently limited (Shrivastava, Shrivastav 2022). Distributed Tracing Modern-day microservices-architecture-based applications are made up of distributed systems that are exceedingly complex to design, and monitoring and debugging them becomes even more complicated. Due to the large number of microservices involved in an application that spans multiple development teams, systems, and infrastructures, even a single request involves a complex network of communication. While this complex distributed system enables a scalable, efficient, and reliable system, it also makes system observability more challenging to achieve, thereby creating issues with troubleshooting. Distributed tracing helps us overcome this observability challenge by using a request-centric view. As a request is processed by the components of a distributed system, distributed tracing captures the detailed execution of the request and its causally related actions across the system's components (Shkuro 2019). Load Balancing Load balancing is the method used to utilize resources optimally and to ensure smooth operational performance. In order to be efficient and scalable, more than one instance of a service is used, and the incoming requests are distributed across these instances for a smooth process flow. In Kubernetes, load balancing algorithms are implemented in a more effective manner using a service mesh, which is based on recorded metrics such as latency. Service meshes mainly manage the traffic between services on the network, ensuring that inter-service communications are safe and reliable by enabling the services to detect and communicate with each other. The use of a service mesh improves observability and aids in monitoring highly distributed systems. Security Each service must be secured individually, and the communication between services must be secure. In addition, there needs to be a centralized way to manage access controls and authentication across all services. One of the most popular ways for securing microservices is to use API gateways, which act as proxies between the clients and the microservices. API gateways can perform authentication and authorization checks, rate limiting, and traffic management. Service Versioning The deployment of a microservice version update often leads to unexpected issues and breaking errors between the new version of the microservice and other microservices in the system, or even external clients using that microservice. While the team deploying the new version attempts to mitigate and reduce these breaks, multiple versions of the same microservice can be run simultaneously, thereby allowing requests to be routed to the appropriate version of the microservice. This is done using API versioning for API contracts. Communication Patterns Communication between microservices can be designed by using two main patterns: synchronous and asynchronous. In Figure 1, we see a basic overview of these communication patterns along with their respective implementation styles and choices. Figure 1. Synchronous and asynchronous communication with common implementation technologies Synchronous Pattern Synchronous communication between microservices is one-to-one communication. The microservice that generates the request is blocked until a response is received from the other service. This is done using HTTP requests or gRPC — a high-performance remote procedure call (RPC) framework. In synchronous communication, the microservices are tightly coupled, which is advantageous for less distributed architectures where communication happens in real time, thereby reducing the complexity of debugging (Newman 2021). Figure 2. Synchronous communication depicting the request-response model The following table shows a comparison between technologies that are commonly used to implement the synchronous communication pattern. Table 1. REST vs. gRPC vs. GraphQL REST gRPC GraphQL Architectural principles Uses a stateless client-server architecture; relies on URIs and HTTP methods for a layered system with a uniform interface Uses the client-server method of remote procedure call; methods are directly called by the client and behave like local methods, although they are on the server side Uses client-driven architecture principles; relies on queries, mutations, and subscriptions via APIs to request, modify, and update data from/on the server HTTP methods POST, GET, PUT, DELETE Custom methods POST Payload data structure to send/receive data JSON- and XML-based payload Protocol Buffers-based serialized payloads JSON-based payloads Request/response caching Natively supported on client and server side Unsupported by default Supported but complex as all requests have a common endpoint Code generation Natively unsupported; requires third-party tools like Swagger Natively supported Natively unsupported; requires third-party tools like GraphQL code generator Asynchronous Pattern In asynchronous communication, as opposed to synchronous, the microservice that initiates the request is not blocked until the response is received. It can proceed with other processes without receiving a response from the microservice it sends the request to. In the case of a more complex distributed microservices architecture, where the services are not tightly coupled, asynchronous message-based communication is more advantageous as it improves scalability and enables continued background operations without affecting critical processes (Newman 2021). Figure 3. Asynchronous communication Event-Driven Communication The event-driven communication pattern leverages events to facilitate communication between microservices. Rather than sending a request, microservices generate events without any knowledge of the other microservices' intents. These events can then be used by other microservices as required. The event-driven pattern is asynchronous communication as the microservices listening to these events have their own processes to execute. The principle behind events is entirely different from the request-response model. The microservice emitting the event leaves the recipient fully responsible for handling the event, while the microservice itself has no idea about the consequences of the generated event. This approach enables loose coupling between microservices (Newman 2021). Figure 4. Producers emit events that some consumers subscribe to Common Data Communication through common data is asynchronous in nature and is achieved by having a microservice store data at a specific location where another microservice can then access that data. The data's location must be persistent storage, such as data lakes or data warehouses. Although common data is frequently used as a method of communication between microservices, it is often not considered a communication protocol because the coupling between microservices is not always observable when it is used. This communication style finds its best use case in situations that involve large volumes of data as a common data location prevents redundancy, makes data processing more efficient, and is easily scalable (Newman 2021). Figure 5. An example of communication through common data Request-Response Communication The request-response communication model is similar to the synchronous communication that was previously discussed — where a microservice provides a request to another microservice and has to await a response. Along with the previously discussed protocols (HTTP, gPRC, etc.), message queues are used as well. Request-response is implemented as one of the following two methods: Blocking synchronous – Microservice A opens a network connection and sends a request to Microservice B along this connection. The established connection stays open while Microservice A waits for Microservice B to respond. Non-blocking asynchronous – Microservice A sends a request to Microservice B, and Microservice Bneeds to know implicitly where to route the response. Also, message queues can be used; they provide an added benefit of buffering multiple requests in the queue to await processing. This method is helpful in situations where the rate of requests received exceeds the rate of handling these requests. Rather than trying to handle more requests than its capacity, the microservice can take its time generating a response before moving on to handle the next request (Newman 2021). Figure 6. An example of request-response non-blocking asynchronous communication Conclusion In recent years, we have observed a paradigm shift from designing large, clunky, monolithic applications that are complex to scale and maintain to using microservices-based architectures that enable the design of distributed applications — ones that can integrate multiple communication patterns and protocols across systems. These complex distributed systems can be developed, deployed, scaled, and maintained independently by different teams with fewer conflicts, resulting in a more robust, reliable, and resilient application. Using the most optimal communication pattern and protocol for the exact operation that a microservice must achieve is a crucial task and has a huge impact on the functionality and performance of an application. The aim is to make the communication between microservices as seamless as possible to establish an efficient system. In-depth knowledge regarding the available communication patterns and protocols is an essential aspect of modern-day cloud-based application design that is not only dynamic but also highly competitive with multiple contenders providing identical applications and services. Speed, scalability, efficiency, security, and other additional features are often crucial in determining the overall quality of an application, and proper microservices communication is the backbone to achieving those capabilities. References: Shrivastava, Saurabh. Shrivastav, Neelanjali. 2022. Solutions Architect's Handbook, 2nd Edition. Packt. Shkuro, Yuri. 2019. Mastering Distributed Tracing. Packt. Newman, Sam. 2021. Building Microservices, 2nd Edition. O'Reilly. This is an excerpt from DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices.Read the Free Report
APIs and SDKs are the bridge to an underlying platform, allowing firms to build applications and integrate your platform into their business processes. Building APIs and SDKs that developers love to use is the key to a successful platform strategy, be it for internal teams or external teams. In the following article, I will provide some of the most effective practices I have seen in the industry. I place these four necessary strategies that should be at the heart of any API/SDK program: simplicity, resilience, community building, and continuous improvement. Prioritize Simplicity Simplicity is the most essential factor to consider while designing APIs and SDKs. Firms are more likely to adopt and stay with you if API and SDK usage is intuitive, well-documented, and easy to plug into other projects. Do not over-engineer or overcomplicate APIs/SDKs. Preferring clarity, consistency, and compliance with industry standards, draft intuitive and user-friendly APIs and SDKs. Create endpoints with concise, descriptive naming best practices that accurately convey their purpose. Codify in-house standards on your entire API or SDK, with appropriate naming conventions and design patterns. Align with widely adopted standards and paradigms, such as RESTful principles, appropriate HTTP methods, language-specific conventions, and secure authentication mechanisms, to provide a seamless and familiar experience for developers. Here are a few good examples to consider: API Design Guidelines APIs Design API Standards Style Guide Blindly following a style guide without considering the unique requirements and goals of your platform can lead to suboptimal outcomes. It is important to strike a balance between catering to developers' needs and doing what's right for the long-term success and viability of your platform. While it might be tempting to fulfill every feature request from your users, you must make hard choices to prioritize the health and maintainability of your platform (the adage applies: "Put on your own oxygen mask first before assisting others"). Nothing erodes trust like a platform that lacks stability, and security or cannot scale, so work hard to find the right balance between a good developer-friendly experience while ensuring the aforementioned criteria are not in peril. Designing for Resilience While designing APIs and SDKs, it is essential to place error handling at its core. To provide a dependable developer experience, a platform needs to have a comprehensive and well-documented error code system that covers a significant range of possible failure scenarios with dozens of unique error codes designed to cover various categories of errors like authentication failures, validation errors, resource not found, rate limiting, and other server-side errors. Furthermore, error messages should not only inform the developer about the nature of an error but offer guidance on how to resolve it. Offer retry mechanisms to developers when dealing with partial failures. Provide them with the means to configure the retry behavior, such as the maximum number of retries and initial retry delay. Additionally, set timeout values to prevent requests to services from hanging or being blocked indefinitely. Allows developers to customize the timeout setting and provides them with a way to gracefully cancel a long-running request. Follow an all-or-nothing approach when it comes to transactional operations. Keep data integrity and consistency in the forefront whenever a batch operation is invoked, either all operations in the batch should succeed or none of them should. The developer should be notified about which items in the batch were successful, and which items were erroneous. Ensure that your APIs and SDKs include robust logging capabilities that can help developers troubleshoot and debug issues. Log relevant information such as request/response details, error messages, and stack traces. Allow developers to configure logging verbosity and opt in/out of logging entirely in production. Define a consistent and clear versioning policy for your APIs and SDKs. Follow semantic versioning. Fostering a Developer Community Building a strong developer community around your APIs and SDKs is critical to drive adoption, educate developers, and promote innovation. Provide comprehensive documentation for your APIs and SDKs that thoroughly covers all they have to offer. Include getting started guides, tutorials, code samples, reference documentation, and more. Build an interactive developer portal that serves as the central hub for all developer-related content. Include features such as API consoles, sandbox environments, and interactive documentation that allow developers to experiment and try out their integrations in a controlled setting. Engage with developers through popular developer platforms, social media, webinars, and in-person workshops. Participate in discussions, answer questions, and provide support for developers who are using your APIs and SDKs. Create an environment where developers can easily provide feedback and help test and improve your offerings. Set up bug trackers, feature requests, and general feedback submission processes. Foster community-driven support by encouraging developers to help each other in forums, establish a community-driven knowledge base, and provide moderation to ensure a positive and inclusive community. Make sure your support team is responsive and knowledgeable, reply to developer questions promptly, and provide value-added responses. Keep a detailed internal knowledge base or a dedicated FAQ section containing solutions to common questions and challenges. This ensures your support and field teams can quickly understand and resolve customer issues, delivering a seamless experience to the developers using your APIs and SDKs. Organize developer events and conferences to gather developers and encourage one-on-one communication. Invite veterans and industry experts to educate and enlighten, and enable developers to present their own projects to learn from one another. Gather feedback, announce features or changes, and bond with your developer community. Growing a thriving developer community ensures you have a supportive environment that cultivates collaboration, education, and innovation, driving your APIs and SDKs to become more popular and successful. Iterate and Improve Develop a structured approach for assessing and ordering the feedback by its effect, urgency, and relationship with your company’s objectives. Regularly consult your development team and stakeholders to review the feedback and determine what changes and features should be implemented into your roadmap. Devote resources and set deadlines to implement the modifications. Ensure your development cycle includes complete testing and quality assurance procedures to uphold the integrity and dependability of your APIs and SDKs. Update your documentation and announce the changes to your developer community. Establish key performance indicators – API adoption rates, developer satisfaction, support ticket response time, for example – to evaluate your changes’ performance. Regularly monitor and assess this data to evaluate the effect of your changes and identify potential improvements. Lastly, build a culture of continuous learning and improvement within your organization. Ensure that your team keeps up with the latest trends in the industry, attends conferences and workshops, and participates in developer communities. Knowledge of the current trends equips you with relevant insights to stay ahead by addressing the current developers’ needs. More importantly, have processes that enable you to iterate and enhance the APIs and SDKs provided. Having a process that can effectively iterate shows developers that you are serious about delivering quality products and can quickly switch to another provider should their expectations be compromised. This way, you build trust and relationships that last, and your platform becomes a reliable and innovative tool that keeps attracting developers in the market. In conclusion, designing developer-friendly APIs and SDKs is a vital element of platform strategy. Prioritize simplicity, resilience, community, and continuous improvement. Remember, developers will only love your platform if they first enjoy using it. Hence, invest in making their experience better, meet their nowadays-changing needs, and enhance their satisfaction. Such actions enable you to get the best out of your platform, introduce more innovations, and thrive in the dynamic technology landscape.
The Advantages of Elastic APM for Observing the Tested Environment My first use of the Elastic Application Performance Monitoring (Elastic APM) solution coincides with projects that were developed based on microservices in 2019 for the projects on which I was responsible for performance testing. At that time (2019) the first versions of Elastic APM were released. I was attracted by the easy installation of agents, the numerous protocols supported by the Java agent (see Elastic supported technologies) including the Apache HttpClient used in JMeter and other languages (Go, .NET, Node.js, PHP, Python, Ruby), and the quality of the dashboard in Kibana for the APM. I found the information displayed in the Kibana APM dashboards to be relevant and not too verbose. The Java agent monitoring is simple but displays essential information on the machine's OS and JVM. The open-source aspect and the free solution for the main functions of the tool were also decisive. I generalize the use of the Elastic APM solution in performance environments for all projects. With Elastic APM, I have the timelines of the different calls and exchanges between web services, the SQL queries executed, the exchange of messages by JMS file, and monitoring. I also have quick access to errors or exceptions thrown in Java applications. Why Integrate Elastic APM in Apache JMeter By adding Java APM Agents to web applications, we find the services called timelines in the Kibana dashboards. However, we remain at a REST API call level mainly, because we do not have the notion of a page. For example, page PAGE01 will make the following API calls: /rest/service1 /rest/service2 /rest/service3 On another page, PAGE02 will make the following calls: /rest/service2 /rest/service4 /rest/service5 /rest/service6 The third page, PAGE03, will make the following calls: /rest/service1 /rest/service2 /rest/service4 In this example, service2 is called on 3 different pages and service4 in 2 pages. If we look in the Kibana dashboard for service2, we will find the union of the calls of the 3 calls corresponding to the 3 pages, but we don't have the notion of a page. We cannot answer "In this page, what is the breakdown of time in the different REST calls," because for a user of the application, the notion of page response time is important. The goal of the jmeter-elastic-apm tool is to add the notion of an existing page in JMeter in the Transaction Controller. This starts in JMeter by creating an APM transaction, and then propagating this transaction identifier (traceparent) with the Elastic agent to an HTTP REST request to web services because the APM Agent recognizes the Apache HttpClient library and can instrument it. In the HTTP request, the APM Agent will add the identifier of the APM transaction to the header of the HTTP request. The headers added are traceparent and elastic-apm-traceparent. We start from the notion of the page in JMeter (Transaction Controller) to go to the HTTP calls of the web application (gestdoc) hosted in Tomcat. In the case of an application composed of multi-web services, we will see in the timeline the different web services called in HTTP(s) or JMS and the time spent in each web service. This is an example of technical architecture for a performance test with Apache JMeter and Elastic APM Agent to test a web application hosted in Apache Tomcat. How the jmeter-elastic-apm Tool Works jmeter-elastic-apm adds Groovy code before a JMeter Transaction Controller to create an APM transaction before a page. In the JMeter Transaction Controller, we find HTTP samplers that make REST HTTP(s) calls to the services. The Elastic APM Agent automatically adds a new traceparent header containing the identifier of the APM transaction because it recognizes the Apache HttpClient of the HTTP sampler. The Groovy code terminates the APM transaction to indicate the end of the page. The jmeter-elastic-apm tool automates the addition of Groovy code before and after the JMeter Transaction Controller. The jmeter-elastic-apm tool is open source on GitHub (see link in the Conclusion section of this article). This JMeter script is simple with 3 pages in 3 JMeter Transaction Controllers. After launching the jmeter-elastic-apm action ADD tool, the JMeter Transaction Controllers are surrounded by Groovy code to create an APM transaction before the JMeter Transaction Controller and close the APM transaction after the JMeter Transaction Controller. In the “groovy begin transaction apm” sampler, the Groovy code calls the Elastic APM API (simplified version): Groovy Transaction transaction = ElasticApm.startTransaction(); Scope scope = transaction.activate(); transaction.setName(transactionName); // contains JMeter Transaction Controller Name In the “groovy end transaction apm” sampler, the groovy code calls the ElasticApm API (simplified version): Groovy transaction.end(); Configuring Apache JMeter With the Elastic APM Agent and the APM Library Start Apache JMeter With Elastic APM Agent and Elastic APM API Library Declare the Elastic APM Agent URLto find the APM Agent: Add the ELASTIC APM Agent somewhere in the filesystem (could be in the <JMETER_HOME>\lib but not mandatory). In <JMETER_HOME>\bin, modify the jmeter.bat or setenv.bat. Add Elastic APM configuration like so: Shell set APM_SERVICE_NAME=yourServiceName set APM_ENVIRONMENT=yourEnvironment set APM_SERVER_URL=http://apm_host:8200 set JVM_ARGS=-javaagent:<PATH_TO_AGENT_APM_JAR>\elastic-apm-agent-<version>.jar -Delastic.apm.service_name=%APM_SERVICE_NAME% -Delastic.apm.environment=%APM_ENVIRONMENT% -Delastic.apm.server_urls=%APM_SERVER_URL% 2. Add the Elastic APM library: Add the Elastic APM API library to the <JMETER_HOME>\lib\apm-agent-api-<version>.jar. This library is used by JSR223 Groovy code. Use this URL to find the APM library. Recommendations on the Impact of Adding Elastic APM in JMeter The APM Agent will intercept and modify all HTTP sampler calls, and this information will be stored in Elasticsearch. It is preferable to voluntarily disable the HTTP request of static elements (images, CSS, JavaScript, fonts, etc.) which can generate a large number of requests but are not very useful in analyzing the timeline. In the case of heavy load testing, it's recommended to change the elastic.apm.transaction_sample_rate parameter to only take part of the calls so as not to saturate the APM Server and Elasticsearch. This elastic.apm.transaction_sample_rate parameter can be declared in <JMETER_HOME>\jmeter.bat or setenv.bat but also in a JSR223 sampler with a short Groovy code in a setUp thread group. Groovy code records only 50% samples: Groovy import co.elastic.apm.api.ElasticApm; // update elastic.apm.transaction_sample_rate ElasticApm.setConfig("transaction_sample_rate","0.5"); Conclusion The jmeter-elastic-apm tool allows you to easily integrate the Elastic APM solution into JMeter and add the notion of a page in the timelines of Kibana APM dashboards. Elastic APM + Apache JMeter is an excellent solution for understanding how the environment works during a performance test with simple monitoring, quality dashboards, time breakdown timelines in the different distributed application layers, and the display of exceptions in web services. Over time, the Elastic APM solution only gets better. I strongly recommend it, of course, in a performance testing context, but it also has many advantages in the context of a development environment used for developers or integration used by functional or technical testers. Links Command Line Tool jmeter-elastic-apm JMeter plugin elastic-apm-jmeter-plugin Elastic APM Guides: APM Guide or Application performance monitoring (APM)
A retry mechanism is a critical component of many modern software systems. It allows our system to automatically retry failed operations to recover from transient errors or network outages. By automatically retrying failed operations, retry mechanisms can help software systems recover from unexpected failures and continue functioning correctly. Today, we'll take a look at these topics: What Is a Retry Pattern? What is it for, and why do we need to implement it in our system? When to Retry Your Request Only some requests should be retried. It's important to understand what kind of errors from the downstream service can be retried to avoid problems with business logic. Retry Backoff Period When we retry the request to the downstream service, how long should we wait to send the request again after it fails? How to Retry We'll look at ways to retry from the basic to more complex. What Is a Retry Pattern? Retrying is an act of sending the same request if the request to the downstream service fails. By using a retry pattern, you'll be improving the downstream resiliency aspect of your system. When an error happens when calling a downstream service, our system will try to call it again instead of returning an error to the upstream service. So, why do we need to do it, exactly? Microservices architecture has been gaining popularity in recent decades. While this approach has many benefits, one of the downsides of microservices architecture is introducing network communication between services. Additional network communication leads to the possibility of errors in the network while services are communicating with each other (read Fallacies of distributed computing). Every call to other services has a chance of getting those errors. In addition, whether you're using monolith or microservices architecture, there is a big chance that you still need to call other services that are not within your company's internal network. Calling service within a different network means your request will go through more network layers and have more chance of failure. Other than network errors, you can also get system errors like rate-limit errors, service downtime, and processing timeout. The errors you get may or may not be suitable to be retried. Let's head to the next section to explore it in more detail. When To Retry Your Request Although adding a retry mechanism in your system is generally a good idea, not every request to the downstream service should be retried. As a simple baseline, things you should consider when you want to retry are as follows: Is It a Transient Error? You'll need to consider whether the type of errors you're getting is transient (temporary). For example, you can retry a connection timeout error because it's usually only temporary, but not a bad request error because you need to change the request. Is It a System Error? When you're getting an error message from the downstream service, it can be categorized as either a system error or an application error. System error is generally okay to be retried because your request hasn't been processed by the downstream service yet. On the other hand, an application error usually means that something is wrong with your request, and you should not retry it. For example, if you're getting a bad request error from the downstream service, you'll always get the same error no matter how many times you've retried. Idempotency Even when you're getting an error from the downstream service, there is still a chance it has processed your request. The downstream service could send the error after it has processed the main process, but another sub-process causes errors. Idempotent API means that even if the API gets the same request twice, it will only process the first request. We can achieve it by adding some ID in the request that's unique to the request so the downstream service can determine whether it should process the request. Usually, you can differentiate this with the Request method. GET, DELETE, and PUT are usually idempotent, and POST is not. However, you need to confirm the API's idempotency to the service owner. The Cost of Retrying When you retry your request to the downstream service, there will be additional resource usage. The additional resource usage can be in the form of additional CPU usage, blocked thread, additional memory usage, additional bandwidth usage, etc. You need to consider this, especially if your service expects large traffic. The Implementation Cost of the Retry Mechanism Many programming languages already have a library that implements a retry mechanism, but you still need to determine which request to retry. You can also create your retry mechanism or every system if you want to, but of course, this means that there will be a high implementation cost for the retry mechanism. Note 1: Many libraries have already implemented the retry mechanism gracefully. For example, if you're using the Spring Mongo library in Java Spring Boot and the connection between your apps and MongoDB is severed, it will try to reconnect. Note 2: Some libraries also implement a retry mechanism by default. It's sometimes dangerous because you can be unaware that the library will retry your request. I've also compiled some common errors and whether or not they're suitable for retrying: Let's describe the errors shortly one by one. Connection timeout: Your app failed to connect to the downstream service; hence, the downstream service isn't aware of your request, and you can retry it. Read timeout: The downstream app has processed your request but has not returned any response for a long time. Circuit breaker tripped: This is an error if you use a circuit breaker in your service. You can retry this kind of error because your service hasn't sent its request to the downstream service. 400 - Bad Request: This error means your request to the downstream service was flagged your request as a wrong request after validating it. You shouldn't retry this error because it will always return the same error if the request is the same. 401 - Unauthorized: You need to authorize before sending the request. Whether you can retry this error will depend on the authentication method and the error. But generally, you will always get the same error if your request is the same. 429 - Too many requests: Your request is rate-limited by the downstream service. You can retry this error, although you should confirm with the downstream service's owner how long your request will be rate-limited. 500 - Internal Server Error: This means the downstream service had started processing your request but failed in the middle of it. Usually, it's okay to retry this error. 503 - Service Unavailable: The downstream service is unavailable due to downtime. It is okay to retry this kind of error. Retry Backoff Period When your request fails to the downstream service, your system will need to wait for some time before trying again. This period is called the retry backoff period. Generally, there are three strategies for wait time between calls: Fixed Backoff, Exponential Backoff, and Random Backoff. All three of them have their advantages and disadvantages. Which one you use should depend on your API and service use case. Fixed Backoff Fixed backoff means that every time you retry your request, the delay between requests is always the same. For example, if you do a retry twice with a backoff of 5 seconds, then if the first call fails, the second request will be sent 5 seconds later. If it fails again, the third call will be sent 5 seconds after the failure. A fixed backoff period is suitable for a request coming directly from the user and needs a quick response. If the request is important and you need it to come back ASAP, then you can set the backoff period to none or close to 0. Exponential Backoff When downstream service is having a problem, it doesn't always recover quickly. What you don't want to do when the downstream service is trying to recover is to hit it multiple times in a short interval. Exponential backoff works by adding some additional backoff time every time our service attempts to call the downstream service. For example, we can configure our retry mechanism with a 5-second initial backoff and add two as the multiplier every attempt. This means when our first call to the downstream service fails, our service will wait 5 seconds before the next call. If the second call fails again, the service will wait 10 seconds instead of 5 seconds before the next call. Due to its longer interval nature, exponential backoff is unsuitable for retrying a user request. But it will be perfect for a background process like notification, sending email, or webhook system. Random Backoff Random backoff is a backoff strategy introducing randomness in its backoff interval calculation. Suppose that your service is getting a burst of traffic. Your service then calls a downstream service for every request, and then you get errors from it because the downstream service gets overwhelmed by your request. Your service implements a retry mechanism and will retry the requests in 5 seconds. But there is a problem: when it's time to retry the requests, all of them will be retried at once, and you might get an error from the downstream service again. With the randomness introduced by the random backoff mechanism, you can avoid this. A random backoff strategy will help your service to level the request to the downstream service by introducing a random value for retry. Let's say you configure the retry mechanism with a 5-second interval and two retries. If the first call fails, the second one could be attempted after 500ms; if it fails again, the third one could be attempted after 3.8 seconds. If many requests fail the downstream service, they won't be retried simultaneously. Where To Store the Retry State When doing a retry, you'll need to store the state of the retry somewhere. The state includes how many retries have been made, the request to be retried, and the additional metadata you want to save. Generally, there are three places you can use to store the retry state, which are as follows: Thread Thread is the most common place to store the retry state. If you're using a library with a built-in retry mechanism, it will most likely use the Thread to store the state. The simplest way to do this is to sleep the Thread. Let's see an example in Java: int retryCount = 0; while (retryCount < 3) { try { thirdPartyOutboundService.getData(); } catch (Exception e) { retryCount += 1; Thread.sleep(3000); } } The code above basically sleeps the Thread when getting an exception and calling the process again. While this is simple, it has the disadvantage of blocking the Thread and making other processes unable to use the Thread. This method is suitable for a fixed backoff strategy with a low interval like processes that direct response to the user and need a response as soon as possible. Messaging We could use a popular messaging broker like RabbitMQ (delayed queue) to save a retry state. When you're getting a request from the upstream, and you fail to process it (it can be because of downstream service or not), you can publish the message to the delayed queue to consume it later (depending on your backoff). Using messaging to save the retry state is suitable for a background process request because the upstream service can't directly get the response of the retry process. The advantage of using this approach is that it's usually easy to implement because the broker/library already supports the retry function. Messaging as a storage system of retry state also works well with distributed systems. One problem that can happen is your service suddenly has a problem like downtime when waiting for the next retry. By saving the retry state in the messaging broker, your service can continue the retry after the issue has been resolved. Database The database is the most customizable solution to store the retry state, either by using a persistent storage or an in-memory KV store like Redis. When the request to the downstream service fails, you can save the data in the database and use a cron job to check the database every second or minute to retry failed messages. While this is the most customizable solution, the implementation cost will be very high because you'll need to implement your retry mechanism. You can either create the mechanism in your service with the downside of sacrificing a bit of performance when a retry is happening or make an entirely new service for retry purposes. Takeaways This article has explored what is and what aspects to consider when implementing a retry pattern. You need to know what request and how to retry it. If you do the retry mechanism correctly, you'll help with the user experience and reduced operation of the service you're building. But, if you do it incorrectly, you risk worsening the user experience and business error. You need to understand when the request can be retried and how to retry it so you can implement the mechanism correctly. There is much more. In this article, we've covered the retry pattern. This pattern increases the downstream resiliency aspect of a system, but there is more to the downstream resiliency. We can combine the retry pattern with a timeout (which we explored in this article) and circuit breaker to make our system more resilient to downstream failure.
Salesforce Analytics Query Language (SAQL) is a Salesforce proprietary query language designed for analyzing Salesforce native objects and CRM Analytics datasets. SAQL enables developers to query, transform, and project data to facilitate business insights by customizing the CRM dashboards. SAQL is very similar to SQL (Structured Query Language); however, it is designed to explore data within Salesforce and has its own unique syntax which is somewhat like Pig Latin (pig-ql). You can also use SAQL to implement complex logic while preparing datasets using dataflows and recipes. Key Features Key features of SAQL include the following: It enables users to specify filter conditions, and group and summarize input data streams to create aggregated values to derive actionable insights and analyze trends. SAQL supports conditional statements such as IF-THEN-ELSE and CASE. This feature can be used to execute complex conditions for data filtering and transformation. SAQL DATE and TIME-related functions make it much easier to work with date and time attributes, allowing users to execute time-based analysis, like comparing the data over various time intervals. Supports a variety of data transformation functions to cleanse, format, and typecast data to alter the structure of data to suit the requirements SAQL enables you to create complex calculated fields using existing data fields by applying mathematical, logical, or string functions. SAQL provides seamless integration with the Salesforce objects and CRM Analytics datasets. SAQL queries can be used to design visuals like charts, graphs, and dashboards within the Salesforce CRM Analytics platform. The rest of this article will focus on explaining the fundamentals of writing the SAQL queries, and delve into a few use cases where you can use SAQL to analyze the Salesforce data. Basics of SAQL Typical SAQL queries work like any other ETL tool: queries load the datasets, perform operations/transformations, and create an output data stream to be used in visualization. SAQL statements can run into multiple lines and are concluded with a semicolon. Every line of the query works on a named stream, which can serve as input for any subsequent statements in the same query. The following SAQL query can be used to create a data stream to analyze the opportunities booked in the previous year by month. SQL 1. q = load "OpportunityLineItems"; 2. q = filter q by 'StageName' == "6 - Closed Won" and date('CloseDate_Year', 'CloseDate_Month', 'CloseDate_Day') in ["1 year ago".."1 year ago"]; 3. q = group q by ('CloseDate_Year', 'CloseDate_Month'); 4. q = foreach q generate q.'CloseDate_Year' as 'CloseDate_Year', q.'CloseDate_Month' as 'CloseDate_Month', sum(q.'ExpectedTotal__c') as 'Bookings'; 5. q = order q by ('CloseDate_Year' asc, 'CloseDate_Month' asc); 6. q = limit q 2000; Line Number Description 1 This statement loads the CRM analytics dataset named “OpportunityLineItems” into an input stream q. 2 The input stream q is filtered to look for the opportunities closed won in the previous year. This is similar to the WHERE clause in SQL. 3 The statement focuses on grouping the records by the close date year and month so that we can visualize this data by the months. This is similar to the GROUP BY clause in SQL. 4 Statement 4 is selecting the attributes we want to project from the input stream. Here the expected total is being summed up for each group. 5 Statement 5 is ordering the records by the close of the year and month so that we can create a line chart to visualize this by month. 6 The last statement in the code above focuses on restricting the stream to a limited number of rows. This is mainly used for debugging purposes. Joining Multiple Data Streams The SAQL cogroup function joins input data streams like Salesforce objects or CRM analytics datasets. The data sources being joined should have a related column to facilitate the join. cogroup also supports the execution of both INNER and OUTER joins. For example, if you had two datasets, with one containing sales data and another containing customer data, you could use cogroup to join them based on a common field like customer ID. The resultant data stream contains both fields from both tables. Use Case The following code block can be used for a data stream for NewPipeline and Bookings for the customers. The pipeline built and bookings are coming from two different streams. We can join these two streams by Account Name. SQL q = load "Pipeline_Metric"; q = filter q by 'Source' in ["NewPipeline"]; q = group q by 'AccountName'; q = foreach q generate q.'AccountName' as 'AccountName', sum(ExpectedTotal__c) as 'NewPipeline'; q1 = load "Bookings_Metric"; q1 = filter q1 by 'Source' in ["Bookings"]; q1 = group q1 by 'AccountName'; q1 = foreach q1 generate q1.'AccountName' as 'AccountName', sum(q1.ExpectedTotal__c) as 'Bookings'; q2 = cogroup q by 'AccountName', q1 by 'AccountName'; result = foreach q2 generate q.'AccountName' as 'AccountName', sum(q.'NewPipeline') as 'NewPipeline',sum(q1.'Bookings') as 'Bookings'; You can also use a left outer cogroup to join the right data table with the left. This will result in all the records from the left data stream and all the matching records from the right stream. Use the coalesce function to replace all the null values from the right stream with another value. In the example above, if you want to report all the accounts with or without bookings, you can use the query below. SQL q = load "Pipeline_Metric"; q = filter q by 'Source' in ["NewPipeline"]; q = group q by 'AccountName'; q = foreach q generate q.'AccountName' as 'AccountName', sum(ExpectedTotal__c) as 'NewPipeline'; q1 = load "Bookings_Metric"; q1 = filter q1 by 'Source' in ["Bookings"]; q1 = group q1 by 'AccountName'; q1 = foreach q1 generate q1.'AccountName' as 'AccountName', sum(q1.ExpectedTotal__c) as 'Bookings'; q2 = cogroup q by 'AccountName' left, q1 by 'AccountName'; result = foreach q2 generate q.'AccountName' as 'AccountName', sum(q.'NewPipeline') as 'NewPipeline', coalesce(sum(q1.'Bookings'), 0) as 'Bookings'; Top N Analysis Using Windowing SAQL enables Top N analysis across value groups using the windowing functions within the input data stream. These functionalities are utilized for deriving the moving averages, cumulative totals, and rankings within the groups. You can specify the set of records where you want to execute these calculations using the “over” keyword. SAQL allows you to specify an offset to identify the number of records before and after the selected row. Optionally you can choose to work on all the records within a partition. These records are called windows. Once the set of records is identified for a window, you can apply an aggregation function to all the records within the defined window. Optionally you can create partitions to group the records based on a set of fields and perform aggregate calculations for each partition independently. Use Case The following SAQL code can be used to prepare data for the percentage contribution of new pipelines for each customer to the total pipeline by the region and the ranking of these customers by the region. SQL q = load "Pipeline_Metric"; q = filter q by 'Source' in ["NewPipeline"]; q = group q by ('Region','AccountName'); q = foreach q generate q.'Region' as 'Region',q.'AccountName' as 'AccountName', ((sum('ExpectedTotal__c')/sum(sum('ExpectedTotal__c')) over ([..] partition by 'Region')) * 100) as 'PCT_PipelineContribution', rank() over ([..] partition by ('Region') order by sum('ExpectedTotal__c') desc ) as 'Rank'; q = filter q by 'Rank' <=5; Data Aggregation: Grand Totals and Subtotals With SAQL SAQL offers rollup and grouping functions to aggregate the data streams based on pre-defined groups. While the rollup construct is used with the group by statement, grouping is used as part of foreach statements while projecting the input data stream. The rollup function aggregates the input data stream at various levels of hierarchy allowing you to create calculated fields on summarized datasets at higher levels of granularity. For example, in case you have datasets by the day, rollup can be used to aggregate the results by week, month, or year. The grouping function is used to group data based on specific dimensions or fields in order to segment the data into meaningful subsets for analysis. For example, you might group sales data by product category or region to analyze performance within each group. Use Case Use the code below to prepare data for the total number of accounts and accounts engaged by the region and theater. Also, add the grand total to look at the global numbers and subtotals for both regions and theaters. SQL q = load "ABXLeadandOpportunities_Metric"; q = filter q by 'Source' == "ABX Opportunities" and 'CampaignType' == "Growth Sprints" and 'Territory_Level_01__c' is not null; q = foreach q generate 'Territory_Level_01__c' as 'Territory_Level_01__c','Territory_Level_02__c' as 'Territory_Level_02__c','Territory_Level_03__c' as 'Territory_Level_03__c', q.'AccountName' as 'AccountName',q.'OId' as 'OId','MarketingActionedOppty' as 'MarketingActionedOppty','AccountActionedAcct' as 'AccountActionedAcct','ADRActionedOppty' as 'ADRActionedOppty','AccountActionedADRAcct' as 'AccountActionedADRAcct'; q = group q by rollup ('Territory_Level_01__c', 'Territory_Level_02__c'); q = foreach q generate case when grouping('Territory_Level_01__c') == 1 then "TOTAL" else 'Territory_Level_01__c' end as 'Level1', case when grouping('Territory_Level_02__c') == 1 then "LEVEL1 TOTAL" else 'Territory_Level_02__c' end as 'Level2', unique('AccountName') as 'Total Accounts',unique('AccountActionedAcct') as 'Engaged',((unique('AccountActionedAcct') / unique('AccountName'))) as '% of Engaged'; q = limit q 2000; Filling the Missing Date Fields You can use the fill() function to create a record for missing date, week, month, quarter, and year records in your dataset. This comes very handy when you want to show the result as 0 for these missing days/weeks/months instead of not showing them at all. Use Case The following SAQL code allows you to track the number of tasks for the sales agents by the days of the week. In case the agents are on PTO you want to show 0 tasks. SQL q = load "Tasks_Metric"; q = filter q by 'Source' == "Tasks"; q = filter q by date('MetricDate_Year', 'MetricDate_Month', 'MetricDate_Day') in [dateRange([2024,4,23], [2024,4,30])]; q = group q by ('MetricDate_Year', 'MetricDate_Month', 'MetricDate_Day'); q = foreach q generate q.'MetricDate_Year' as 'MetricDate_Year', q.'MetricDate_Month' as 'MetricDate_Month', q.'MetricDate_Day' as 'MetricDate_Day', unique(q.'Id') as 'Tasks'; q = order q by ('MetricDate_Year' asc, 'MetricDate_Month' asc, 'MetricDate_Day' asc); q = limit q 2000; The code above will be missing two days where there were no tasks created. You can use the code below to fill in the missing days. SQL q = load "Tasks_Metric"; q = filter q by 'Source' == "Tasks"; q = filter q by date('MetricDate_Year', 'MetricDate_Month', 'MetricDate_Day') in [dateRange([2024,4,23], [2024,4,30])]; q = group q by ('MetricDate_Year', 'MetricDate_Month', 'MetricDate_Day'); q = foreach q generate q.'MetricDate_Year' as 'MetricDate_Year', q.'MetricDate_Month' as 'MetricDate_Month', q.'MetricDate_Day' as 'MetricDate_Day', unique(q.'Id') as 'Tasks'; q = fill q by (dateCols=(MetricDate_Year, MetricDate_Month, MetricDate_Day, "Y-M-D")); q = order q by ('MetricDate_Year' asc, 'MetricDate_Month' asc, 'MetricDate_Day' asc); q = limit q 2000; You can also specify the start date and end date to populate the missing records between these dates. Conclusion In the end, SAQL has proven itself as a powerful tool for the Salesforce developer community, empowering them to extract actionable business insights from the CRM datasets using capabilities like filtering, aggregation, windowing, time-analysis, blending, custom calculation, Salesforce integration, and performance optimization. In this article, we have explored various capabilities of this technology and focused on targeted use cases. As a next step, I would recommend continuing your learnings by exploring Salesforce documentation, building your data models using dataflow, and using SAQL capabilities to harness the true potential of Salesforce as a CRM.
Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices. A recent conversation with a fellow staff engineer at a Top 20 technology company revealed that their underlying infrastructure is self-managed and does not leverage cloud-native infrastructure offered by major providers like Amazon, Google, or Microsoft. Hearing this information took me a minute to comprehend given how this conflicts with my core focus on leveraging frameworks, products, and services for everything that doesn't impact intellectual property value. While I understand the pride of a Top 20 technology company not wanting to contribute to the success of another leading technology company, I began to wonder just how successful they could be if they utilized a cloud-native approach. That also made me wonder how many other companies have yet to adopt a cloud-native approach… and the impact it is having on their APIs. Why Cloud? Why Now? For the last 10 years, I have been focused on delivering cloud-native API services for my projects. While cloud adoption continues to gain momentum, a decent percentage of corporations and technology providers still utilize traditional on-premises designs. According to The Cloud in 2021: Adoption Continues report by O'Reilly Media, Figure 1 provides a summary of the state of cloud adoption in December 2021. Figure 1. Cloud technology usage Image adapted from The Cloud in 2021: Adoption Continues, O'Reilly Media Since the total percentages noted in Figure 1 exceed 100%, the underlying assumption is that it is common for respondents to maintain both a cloud and on-premises design. However, for those who are late to enter the cloud native game, I wanted to touch on some common benefits that are recognized with cloud adoption: Focus on delivering or enhancing laser-focused APIs — stop worrying about and managing on-premises infrastructure. Scale your APIs up (and down) as needed to match demand — this is a primary use case for cloud adoption. Reduce risk by expanding your API presence — leverage availability zones, regions, and countries. Describe the supporting API infrastructure as code (IaC) — faster recovery and expandability into new target locations. Making the transition toward cloud native has become easier than ever, with the major providers offering free or discounted trial periods. Additionally, smaller platform-as-a-service (PaaS) providers like Heroku and Render provide solutions that allow teams to focus on their products and services and not worry about the underlying infrastructure design. The Cloud Native Impact on Your API Since this Trend Report is focused on modern API management, I wanted to focus on a few of the benefits that cloud native can have on APIs. Availability and Latency Objectives When providing APIs for your consumers to consume, the concept of service-level agreements (SLAs) is a common onboarding discussion topic. This is basically where expectations are put into easy-to-understand wording that becomes a binding contract between the API provider and the consumer. Failure to meet these expectations can result in fees and, in some cases, legal action. API service providers often take things a step further by establishing service-level objectives (SLOs) that are even more stringent. The goal here is to establish monitors and alerts to remediate issues before they breach contractual SLAs. But what happens when the SLOs and SLAs struggle to be met? This is where the primary cloud native use case can assist. If the increase in latency is due to hardware limitations, the service can be scaled up vertically (by increasing the hardware) or horizontally (by adding more instances). If the increase in latency is driven by geographical location, introducing service instances in closer regions is something cloud native providers can provide to remedy this scenario. API Management As your API infrastructure expands, a cloud-native design provides the necessary tooling to ease supportability and manageability efforts. From an infrastructure perspective, the underlying definition of the service is defined using an IaC approach, allowing the service itself to become defined in a single location. As updates are made to that base design, those changes can be rolled out to each target service instance, avoiding any drift between service instances. From an API management perspective, cloud native providers include the necessary tooling to manage the APIs from a usage perspective. Here, API keys can be established, which offer the ability to impose thresholds on requests that can be made or features that align with service subscription levels. Cloud Native !== Utopia While APIs flourish in cloud native implementations, it is important to recognize that a cloud-native approach is not without its own set of challenges. Cloud Cost Management CloudZero's The State Of Cloud Cost Intelligence 2022 report concluded that only 40% of respondents indicated that their cloud costs were at an expected level as noted in Figure 2. Figure 2. Cloud native cost realities Image adapted from The State Of Cloud Cost Intelligence, CloudZero This means that 60% of respondents are dealing with higher-than-expected cloud costs, which ultimately impact an organization's ability to meet planned objectives. Cloud native spending can often be remediated by adopting the following strategies: Require team-based tags or cloud accounts to help understand levels of spending at a finer grain. Focus on storage buckets and database backups to understand if the cost is in line with the value. Engage a cloud business partner that specializes in cloud spending analysis. Account Takeover The concept of accounts becoming "hacked" is prevalent in social media. At times, I feel like my social media feed contains more "my account was hacked" messages than the casual updates I was tuning in to read. Believe it or not, the concept of account takeover is becoming a common fear for cloud native adopters. Imagine starting your day only to realize you no longer have access to any of your cloud-native services. Soon thereafter, your customers begin to flood your support lines to ask what is going on… and where the data they were expecting to see with each API call is. Another potential consequence is that the APIs are shut down completely, forcing customers to seek out competing APIs. Remember, your account protection is only as strong as your weakest link. Make sure to employ everything possible to protect your account and move away from simple username + password account protection. Disaster Recovery It is also important to recognize that cloud native is not a replacement for maintaining a strong disaster recovery posture. Understand the impact of availability zone and region-wide outages — both are expected to happen. Plan to implement immutable backups — avoid relying on traditional backups and snapshots. Leverage IaC to establish all aspects of cloud native — and test it often. Alternative Flows Exist While a cloud-native approach provides an excellent landscape to help your business and partnerships be successful, there are likely use cases that present themselves as alternative flows for cloud native adoption: Regulatory requirements for a given service can often present themselves as an alternative flow and not be a candidate for cloud native adoption. Point of presence requirements can also become a blocker for cloud native adoption when the closest cloud-native location is not close enough to meet the established SLAs and SLOs. On the Other Side of API Cloud Adoption By adopting a cloud-native approach, it is possible to extend an API across multiple availability zones and geographical regions within a given point of presence. Figure 3. Multi-region cloud native adoption In Figure 3, each region contains an API service instance in three different geographical regions. Additionally, each region contains an API service instance running in three different availability zones — each with its own network and power source. In this example, there are nine distinct instances running across the United States. By introducing a global common name, consumers always receive a service response from the least-latent and available service instance. This approach easily allows for entire regions to be taken offline for disaster recovery validation without any interruptions of service at the consumer level. Conclusion Readers familiar with my work may recall that I have been focused on the following mission statement, which I feel can apply to any IT professional: Focus your time on delivering features/functionality that extend the value of your intellectual property. Leverage frameworks, products, and services for everything else. —John Vester When I think about my conversion with the staff engineer at the Top 20 tech company, I wonder how much more successful his team would be without having to worry about the underlying infrastructure being managed with their on-premises approach. While the other side of cloud native is not without challenges, it does adhere to my mission statement. As a result, projects that I have worked on for the last 10 years have been able to remain focused on meeting the needs of API consumers while staying in line with corporate objectives. From an API perspective, cloud native offers additional ways to adhere to my personal mission statement by describing everything related to the service using IaC and leveraging built-in tooling to manage the APIs across different availability zones and regions. Have a really great day! This is an excerpt from DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices.Read the Free Report
Do you want to learn how to create Tweets from a Java application using the X (Twitter) API v2? This blog will show you in a step-by-step guide how to do so. Enjoy! Introduction X (Twitter) provides an API that allows you to interact with your account from an application. Currently, two versions exist. In this blog, you will use the most recent X API v2. Although a lot of information can be found on how to set up your environment and how to interact with the API, it took me quite some time to do so from within a Java application. In this blog, you will learn how to set up your account and how you can create tweets from a Java application. The sources for this blog can be found on GitHub. Prerequisites Prerequisites for this blog are: Basic knowledge of Java, Java 21 is used; An X account; A website you own (not mandatory, but for security reasons the better). Set up Developer Account The first thing to do is to set up a developer account. Navigate to the sign-up page. Beware that there exist multiple types of accounts: free account, basic account, pro account, and enterprise account. Scroll down all the way to the bottom of the page and choose to create a free account by clicking the button Sign up Free Account. You will need to describe your use case using at least 250 characters. After signing up, you end up in the developer portal. Create Project and App With the Free tier, you can create one Project and one App. Create the Project and the App. Authentication As you are going to create Tweets for a user, you will need to set up the authentication using OAuth 2.0 Authorization Code Flow with PKCE. However, it is important that you have configured your App correctly. Navigate to your App in the developer portal and click the Edit button in the User authentication settings. Different sections are available here where you are required to add information and to choose options. App Permissions These permissions enable OAuth 1.0a Authentication. It is confusing that you need to check one of these bullets because OAuth 1.0a Authentication will not be used in your use case. However, because you will create a tweet, select Read and Write, just to be sure. Type of App The type of App will enable OAuth 2.0 Authentication, this is the one you will use. You will invoke the API from an application. Therefore, choose a Web App, Automated App, or Bot. App Info In the App info section, you need to provide a Callback URI and a Website URL. The Callback URI is important as you will see later on in the next paragraphs. Fill in the URL of your website. You can use any URL, but the Callback URI will be used to provide you with an access token, so it is better to use the URL of a website you own. Click the Save button to save your changes. A Client ID and Client Secret are generated and save them. Create Tweet Everything is set up in the developer portal. Now it is time to create the Java application in order to be able to create a Tweet. Twitter API Client Library for Java In order to create the tweet, you will make use of the Twitter API Client Library for Java. Beware that, at the time of writing, the library is still in beta. The library also only supports the X API v2 endpoints, but that is exactly the endpoints you will be using, so that is not a problem. Add the dependency to the pom file: XML <dependency> <groupId>com.twitter</groupId> <artifactId>twitter-api-java-sdk</artifactId> <version>2.0.3</version> </dependency> Authorization In order to be able to create tweets on behalf of your account, you need to authorize the App. The source code below is based on the example provided in the SDK. You need the Client ID and Client Secret you saved before. If you lost them, you can generate a new secret in the developer portal. Navigate to your App, and click the Keys and Tokens tab. Scroll down retrieve the Client ID and generate a new Client Secret. The main method executes the following steps: Set the correct credentials as environment variables: TWITTER_OAUTH2_CLIENT_ID: the OAuth 2.0 client ID TWITTER_OAUTH2_CLIENT_SECRET: the Oauth 2.0 Client Secret TWITTER_OAUTH2_ACCESS_TOKEN: leave it blank TWITTER_OAUTH2_REFRESH_TOKEN: leave it blank Authorize the App and retrieve an access and refresh token. Set the newly received access and refresh token in the credentials object. Call the X API in order to create the tweet. Java public static void main(String[] args) { TwitterCredentialsOAuth2 credentials = new TwitterCredentialsOAuth2(System.getenv("TWITTER_OAUTH2_CLIENT_ID"), System.getenv("TWITTER_OAUTH2_CLIENT_SECRET"), System.getenv("TWITTER_OAUTH2_ACCESS_TOKEN"), System.getenv("TWITTER_OAUTH2_REFRESH_TOKEN")); OAuth2AccessToken accessToken = getAccessToken(credentials); if (accessToken == null) { return; } // Setting the access & refresh tokens into TwitterCredentialsOAuth2 credentials.setTwitterOauth2AccessToken(accessToken.getAccessToken()); credentials.setTwitterOauth2RefreshToken(accessToken.getRefreshToken()); callApi(credentials); } The getAccessToken method executes the following steps: Creates a Twitter service object: Set the Callback URI to the one you specified in the developer portal. Set the scopes (what is allowed) you want to authorize. By using offline.access, you will receive a refresh token which allows you to retrieve a new access token without prompting the user via the refresh token flow. This means that you can continuously create tweets without the need of user interaction. An authorization URL is provided to you where you will authorize the App for the requested scopes. You are redirected to the Callback URI and in the URL the authorization code will be visible. The getAccessToken method waits until you copy the authorization code and hit enter. The access token and refresh token are printed to the console and returned from the method. Java private static OAuth2AccessToken getAccessToken(TwitterCredentialsOAuth2 credentials) { TwitterOAuth20Service service = new TwitterOAuth20Service( credentials.getTwitterOauth2ClientId(), credentials.getTwitterOAuth2ClientSecret(), "<Fill in your Callback URI as configured in your X App in the developer portal>", "offline.access tweet.read users.read tweet.write"); OAuth2AccessToken accessToken = null; try { final Scanner in = new Scanner(System.in, "UTF-8"); System.out.println("Fetching the Authorization URL..."); final String secretState = "state"; PKCE pkce = new PKCE(); pkce.setCodeChallenge("challenge"); pkce.setCodeChallengeMethod(PKCECodeChallengeMethod.PLAIN); pkce.setCodeVerifier("challenge"); String authorizationUrl = service.getAuthorizationUrl(pkce, secretState); System.out.println("Go to the Authorization URL and authorize your App:\n" + authorizationUrl + "\nAfter that paste the authorization code here\n>>"); final String code = in.nextLine(); System.out.println("\nTrading the Authorization Code for an Access Token..."); accessToken = service.getAccessToken(pkce, code); System.out.println("Access token: " + accessToken.getAccessToken()); System.out.println("Refresh token: " + accessToken.getRefreshToken()); } catch (Exception e) { System.err.println("Error while getting the access token:\n " + e); e.printStackTrace(); } return accessToken; } Now that you authorized the App, you are able to see that you have done so in your X settings. Navigate to Settings and Privacy in your X account. Navigate to Security and account access. Navigate to Apps and sessions. Navigate to Connected apps. Here you will find the App you authorized and which authorizations it has. The callApi method executes the following steps: Create a TwitterApi instance with the provided credentials. Create a TweetRequest. Create the Tweet. Java private static void callApi(TwitterCredentialsOAuth2 credentials) { TwitterApi apiInstance = new TwitterApi(credentials); TweetCreateRequest tweetCreateRequest = new TweetCreateRequest(); // TweetCreateRequest | tweetCreateRequest.setText("Hello World!"); try { TweetCreateResponse result = apiInstance.tweets().createTweet(tweetCreateRequest) .execute(); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling TweetsApi#createTweet"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } Add an sdk.properties file to the root of the repository, otherwise, an Exception will be thrown (the Exception is not blocking, but spoils the output). If everything went well, you now have created your first Tweet! Obtain New Access Token You only need to execute the source code above once. The retrieved access token, however, will only stay valid for two hours. After that time (or earlier), you need to retrieve a new access token using the refresh token. The source code below is based on the example provided in the SDK. The main method executes the following steps: Set the credentials including the access and refresh token you obtained in the previous sections. Add a callback to the TwitterApi instance in order to retrieve a new access and refresh token. Request to refresh the token, the callback method MainToken will set the new tokens and again a Tweet can be created. Java public static void main(String[] args) { TwitterApi apiInstance = new TwitterApi(new TwitterCredentialsOAuth2(System.getenv("TWITTER_OAUTH2_CLIENT_ID"), System.getenv("TWITTER_OAUTH2_CLIENT_SECRET"), System.getenv("TWITTER_OAUTH2_ACCESS_TOKEN"), System.getenv("TWITTER_OAUTH2_REFRESH_TOKEN"))); apiInstance.addCallback(new MaintainToken()); try { apiInstance.refreshToken(); } catch (Exception e) { System.err.println("Error while trying to refresh existing token : " + e); e.printStackTrace(); return; } callApi(apiInstance); } The MaintainToken method only sets the new tokens. Java class MaintainToken implements ApiClientCallback { @Override public void onAfterRefreshToken(OAuth2AccessToken accessToken) { System.out.println("access: " + accessToken.getAccessToken()); System.out.println("refresh: " + accessToken.getRefreshToken()); } } Conclusion In this blog, you learned how to configure an App in the developer portal. You learned how to authorize your App from a Java application and how to create a Tweet.
I recently read 6 Ways To Pass Parameters to Spring REST API. Though the title is a bit misleading, as it's unrelated to REST, it does an excellent job listing all ways to send parameters to a Spring application. I want to do the same for Apache APISIX; it's beneficial when you write a custom plugin. General Setup The general setup uses Docker Compose and static configuration. I'll have one plugin per way to pass parameters. YAML services: httpbin: image: kennethreitz/httpbin #1 apisix: image: apache/apisix:3.9.0-debian volumes: - ./apisix/conf/config.yml:/usr/local/apisix/conf/config.yaml:ro - ./apisix/conf/apisix.yml:/usr/local/apisix/conf/apisix.yaml:ro #2 - ./apisix/plugins:/opt/apisix/plugins:ro #3 ports: - "9080:9080" Local httpbin for more reliable results and less outbound network traffic Static configuration file Plugins folder, one file per plugin YAML deployment: role: data_plane role_data_plane: config_provider: yaml #1 apisix: extra_lua_path: /opt/?.lua #2 plugins: - proxy-rewrite #3 - path-variables #4 # ... Set static configuration Use every Lua file under /opt/apisix/plugins as a plugin Regular plugin Custom plugin, one per alternative Path Variables Path variables are a straightforward way to pass data. Their main issue is that they are limited to simple values, e.g., /links/{n}/{offset}. The naive approach is to write the following Lua code: Lua local core = require("apisix.core") function _M.access(_, ctx) local captures, _ = ngx.re.match(ctx.var.uri, '/path/(.*)/(.*)') --1-2 for k, v in pairs(captures) do core.log.warn('Order-Value pair: ', k, '=', v) end end APISIX stores the URI in ctx.var.uri Nginx offers a regular expression API Let's try: Shell curl localhost:9080/path/15/3 The log displays: Plain Text Order-Value pair: 0=/path/15/3 Order-Value pair: 1=15 Order-Value pair: 2=3 I didn't manage errors, though. Alternatively, we can rely on Apache APISIX features: a specific router. The default router, radixtree_host_uri, uses both the host and the URI to match requests. radixtree_uri_with_parameter lets go of the host part but also matches parameters. YAML apisix: extra_lua_path: /opt/?.lua router: http: radixtree_uri_with_parameter We need to update the route: YAML routes: - path-variables - uri: /path/:n/:offset #1 upstream_id: 1 plugins: path-variables: ~ Store n and offset in the context, under ctx.curr_req_matched We keep the plugin just to log the path variables: Lua function _M.access(_, ctx) core.log.warn('n: ', ctx.curr_req_matched.n, ', offset: ', ctx.curr_req_matched.offset) end The result is as expected with the same request as above: Plain Text n: 15, offset: 3 Query Parameters Query parameters are another regular way to pass data. Like path variables, you can only pass simple values, e.g., /?foo=bar. The Lua code doesn't require regular expressions: Lua local core = require("apisix.core") function _M.access(_, _) local args, _ = ngx.req.get_uri_args() for k, v in pairs(args) do core.log.warn('Order-Value pair: ', k, '=', v) end end Let's try: Shell curl localhost:9080/query\?foo=one\&bar=three The log displays: Plain Text Key-Value pair: bar=three Key-Value pair: foo=one Remember that query parameters have no order. Our code contains an issue, though. The ngx.req.get_uri_args() accepts parameters. Remember that the client can pass a query parameter multiple times with different values, e.g., ?foo=one&foo=two? The first parameter is the maximum number of values returned for a single query parameter. To avoid ignoring value, we should set it to 0, i.e., unbounded. Since every plugin designer must remember it, we can add the result to the context for other plugins down the chain. The updated code looks like this: Lua local core = require("apisix.core") function _M.get_uri_args(ctx) if not ctx then ctx = ngx.ctx.api_ctx end if not ctx.req_uri_args then local args, _ = ngx.req.get_uri_args(0) ctx.req_uri_args = args end return ctx.req_uri_args end function _M.access(_, ctx) for k, v in pairs(ctx.req_uri_args) do core.log.warn('Key-Value pair: ', k, '=', v) end end Request Headers Request headers are another way to pass parameters. While they generally only contain simple values, you can also use them to send structured values, e.g., JSON. Depending on your requirement, APISIX can list all request headers or a specific one. Here, I get all of them: Lua local core = require("apisix.core") function _M.access(_, _) local headers = core.request.headers() for k, v in pairs(headers) do core.log.warn('Key-Value pair: ', k, '=', v) end end We test with a simple request: Shell curl -H 'foo: 1' -H 'bar: two' localhost:9080/headers And we got more than we expected because curl added default headers: Plain Text Key-Value pair: user-agent=curl/8.4.0 Key-Value pair: bar=two Key-Value pair: foo=1 Key-Value pair: host=localhost:9080 Key-Value pair: accept=*/* Request Body Setting a request body is the usual way to send structured data, e.g, JSON. Nginx offers a simple API to collect such data. Lua local core = require("apisix.core") function _M.access(_, _) local args = core.request.get_post_args() --1 local body = next(args, nil) --2 core.log.warn('Body: ', body) end Access the body as a regular Lua table A table is necessary in case of multipart payloads, e.g., file uploads. Here, we assume there's a single arg, the content body. It's time to test: Shell curl localhost:9080/body -X POST -d '{ "foo": 1, "bar": { "baz": "two" } }' The result is as expected: JSON Body: { "foo": 1, "bar": { "baz": "two" } } Cookies Last but not least, we can send parameters via cookies. The difference with previous alternatives is that cookies persist on the client side, and the browser sends them with each request. On the Lua side, we need to know the cookie name instead of listing all query parameters or headers. Lua local core = require("apisix.core") function _M.access(_, ctx) local foo = ctx.var.cookie_foo --1 core.log.warn('Cookie value: ', foo) end The cookie is named foo and is case-insensitive Let's test: Shell curl --cookie "foo=Bar" localhost:9080/cookies The result is correct: Plain Text Cookie value: Bar Summary In this post, we listed five alternatives to pass parameters server-side and explained how to access them on Apache APISIX. Here's the API summary: Alternative Source API Path variable APISIX Router Use the radixtree_uri_with_parameter router Query parameter Nginx ngx.req.get_uri_args(0) Request header APISIX core lib core.request.headers() Request body APISIX core lib core.request.get_post_args() Cookie Method context parameter ctx.var.cookie_ Thanks a lot to Zeping Bai for his review and explanations. The complete source code for this post can be found on GitHub. To Go Further 6 Ways To Pass Parameters to Spring REST API How to Build an Apache APISIX Plugin From 0 to 1?
John Vester
Staff Engineer,
Marqeta
Alexey Shepelev
Senior Full-stack Developer,
BetterUp
Saurabh Dashora
Founder,
ProgressiveCoder