Leader Election¶

Overview¶

Leader election lets you run several Workloads of the same Application at the edge, with only one of them acting at a time. If that Workload fails, another one takes over.

Without it, every Workload of an Application is a single point of failure. The kelvin.ha module in the Kelvin SDK does the election for you, so your code only asks "am I the leader?" and does not need its own polling or lease logic.

All the Workloads run the full Application. The one that is elected is the leader and does the work. The others are followers and wait to take over.

How It Works¶

• Each Workload checks in with the Leader Election API of its Cluster. The check-in is also the heartbeat that keeps the Workload in the election
• The API answers with who the leader is and for how long the lease lasts
• A Workload is the leader only while its lease has not expired
• The SDK does the check-ins in the background, about three times for each lease, so two missed requests do not cost the leadership

When to Use Leader Election¶

Use it when you run more than one Workload of the same Application for redundancy, and only one of them should publish, write or act at a time.

When a Workload is promoted, you can resume the work, for example resume publishing. When it is demoted, you can stop, for example drain a queue.

app.yaml Setup¶

There is nothing to declare in the app.yaml. The identity, the election and the address of the Leader Election API come from environment variables that the platform sets in your Workload. You do not need to declare or read them.

Deploy Time Setup¶

Deploy several Workloads of the same Application, as you do today. See Deploy Workload.

Python Example¶

Create a LeaderElection, start it, and do the work only while is_leader is True.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

import time

from kelvin.ha import LeaderElection

leader = LeaderElection()
leader.start()

while True:
    if leader.is_leader:
        publish()
    time.sleep(1)

• Check is_leader for each unit of work. Do not store the result, because it can change at any time
• Do not block the start of your Application waiting to be the leader. Every Workload runs the full Application, so a follower can take over straight away

asyncio Applications¶

AsyncLeaderElection has the same options as a task on your event loop. You await its start() and stop().

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

import asyncio

from kelvin.application import KelvinApp
from kelvin.ha import AsyncLeaderElection

app = KelvinApp()
leader = AsyncLeaderElection()


@app.timer(interval=5)
async def publish() -> None:
    if not leader.is_leader:
        return
    ...  # publish


async def main() -> None:
    await leader.start()
    try:
        await app.run_forever()
    finally:
        await leader.stop()


if __name__ == "__main__":
    asyncio.run(main())

To wait until the Workload is the leader, use await leader.wait_until_leader().

Promotion and Demotion¶

Use the on_promoted and on_demoted hooks to do something when the Workload becomes the leader or stops being the leader, for example to resume publishing or to drain a queue.

You can set them by assigning a function.

1
2
3
4
5
6

from kelvin.ha import LeaderElection

leader = LeaderElection()
leader.on_promoted = resume_publishing
leader.on_demoted = drain_queues
leader.start()

Or with decorators.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

from kelvin.ha import LeaderElection

leader = LeaderElection()


@leader.on_promoted
def resume() -> None:
    publisher.resume()


@leader.on_demoted
def drain() -> None:
    publisher.drain()


leader.start()

• The hooks fire only when the state changes. A lease renewal does not fire on_promoted again
• If a hook raises an error, the error is logged and the election continues
• LeaderElection accepts normal functions only. An async def is logged as an error and is not run
• AsyncLeaderElection accepts normal and async functions, with decorators or by assignment

1
2
3
4
5
6
7
8

from kelvin.ha import AsyncLeaderElection

leader = AsyncLeaderElection()


@leader.on_demoted
async def drain() -> None:
    await publisher.drain()

Scope¶

By default, all the Workloads of an Application compete for the same leadership, the Application itself, krn:app:<app>.

If the default is not right, pass another resource. For example, to let different versions of an Application, such as blue and green, elect their leaders independently:

1
2
3
4

from kelvin.ha import LeaderElection
from kelvin.krn import KRNAppVersion

leader = LeaderElection(resource=KRNAppVersion(app="my-app", version="1.2.0"))

The resource must be a KRN. See the KRN reference.

Current Leader¶

current_leader shows who the leader is, even when it is not you. A follower can use it to log or show which Workload is active.

1

print(leader.current_leader)

It keeps the last leader it saw. It is not cleared when the Workload is demoted or stopped.

Failover Behavior¶

• Lost connection: If the Application cannot reach the Leader Election API, is_leader becomes False as soon as the lease expires, even if the API never said the Workload was demoted. An isolated Workload is demoted within one lease duration, and the API promotes a Workload that it can reach
• Stopping: stop() demotes the Workload straight away and fires on_demoted, so another Workload takes over within the lease duration. Always call stop() when your Application shuts down. Without it, the Workload keeps its lease until it expires and the takeover is delayed

Example: Publish From One Workload Only¶

An Application publishes a value every 5 seconds. You deploy two Workloads of it for redundancy. Only the leader publishes, and the other one waits.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

import asyncio

from kelvin.application import KelvinApp
from kelvin.ha import AsyncLeaderElection
from kelvin.krn import KRNAssetDataStream
from kelvin.logs import logger
from kelvin.message import Number

app = KelvinApp()
leader = AsyncLeaderElection()


@leader.on_promoted
async def on_promoted() -> None:
    logger.info("This Workload is now the leader")


@leader.on_demoted
async def on_demoted() -> None:
    logger.info("This Workload is no longer the leader")


@app.timer(interval=5)
async def publish_sequence() -> None:
    if not leader.is_leader:
        return
    for asset in app.assets:
        await app.publish(
            Number(resource=KRNAssetDataStream(asset, "sequence"), payload=1)
        )


async def main() -> None:
    await leader.start()
    try:
        await app.run_forever()
    finally:
        await leader.stop()


if __name__ == "__main__":
    asyncio.run(main())

Both Workloads declare the same outputs in the app.yaml. If the leader stops, the other Workload is promoted within the lease duration and starts publishing.

Leader Election Reference¶