Important configuration values

This page describe the most common configuration values for Julie Ops, this values can be set within the topology-builder properties file.

Access control configuration

Configure the access control methodology.

Property: topology.builder.access.control.class Default value: “com.purbon.kafka.topology.roles.SimpleAclsProvider” values:

  • RBAC: “com.purbon.topology.roles.RBACProvider”
  • ACLs: “com.purbon.kafka.topology.roles.SimpleAclsProvider”

RBAC configuration

To configure RBAC, as a user you need to setup the access to your MDS server location, for this you need to setup the user and password to access it. An example configuration looks like this:

topology.builder.mds.server = "http://localhost:8090"
topology.builder.mds.user = "mds"
topology.builder.mds.password = "mds-secret"

A part from that, you need to setup the UUID for each of your clusters. This is one like this:

topology.builder.mds.kafka.cluster.id = "foobar"
topology.builder.mds.schema.registry.cluster.id = "schema-registry-cluster"
topology.builder.mds.kafka.connect.cluster.id = "connect-cluster"

Schema Management

If you plan to manage and deploy schemas with Julie Ops, you must define the url to your Confluent Schema Registry as follows

schema.registry.url = "http://localhost:8081"

Topology Builder backend usage and selection

It is necessary for the topology builder to keep some state, if it does not retrieve state from the cluster (see also “topology.state.cluster.enabled”), for example for situations when the tool needs to decide what ACLs to remove. As well this property is important when the tool does not manage all topics in the cluster, so it is important to know it context.

The default implementation is a File, however is possible ot use other systems. To configure it you can use:

Configure the state management system. Property: topology.builder.state.processor.class Default value: “com.purbon.kafka.topology.backend.FileBackend” values:

  • File: “com.purbon.kafka.topology.backend.FileBackend”
  • Redis: “com.purbon.kafka.topology.backend.RedisBackend”

If you are using redis, you need to extend two other properties to setup the server location:

topology.builder.redis.host = "example.com"
topology.builder.redis.port = 6379

Customize the topic naming convention

A request, not either common, but necessary in some situations is to customize the topic naming convention. For this Julie Ops offers the user the option to set it up using the configuration file.

This future accepts patterns using the jinja template formatting. NOTE: The properties used in the template need to exist in the topology as attributes.

As a user you can customize:

  • Property: topology.topic.prefix.format, to set the full topic naming format.
  • Property: topology.project.prefix.format, to set the project level name format, it should be a subset of the previous one.
  • Property: topology.topic.prefix.separator, to select a custom separator between attributes.

Optimised number of ACLs and RBAC bindings

This property is used to reduce the number of ACLs, or RBAC bindings, created. In the normal operational mode, Julie Ops, will create direct pair of bindings for each user and topic. However for some organisations, it might be enough, to create an optimised list by using prefixed bindings.

Property: topology.acls.optimized Default value: “false”

An example configuration might look like this:

topology.acls.optimized=true

Internal topics prefixes

This is used to avoid deleting topics not controlled by topology builder.

Property: kafka.internal.topic.prefixes Default value: “_”

An example configuration might look like this:

kafka.internal.topic.prefixes.0=_
kafka.internal.topic.prefixes.1=topicPrefixA
kafka.internal.topic.prefixes.2=topicPrefixB

Topology validations

It is possible to define a list of validations to be applied to the topology files.

As a user you can list the validation classes to be applied using the configuration property:

  • Property: topology.validations

This property accepts a list of validation classes available in the class path. Use the fully qualified class name. They will be applied in sequence as defined. You will find included KTB validations in the package ‘com.purbon.kafka.topology.validation’.

An example configuration might look like this:

topology.validations.0=com.purbon.kafka.topology.validation.topology.CamelCaseNameFormatValidation
topology.validations.1=com.purbon.kafka.topology.validation.topic.PartitionNumberValidation

You can also create your own custom validations. The validations must implement one of these interfaces: - com.purbon.kafka.topology.validation.TopologyValidation - com.purbon.kafka.topology.validation.TopicValidation

Prevent ACL for topic creation for connector principal

By default Julie Ops will create the ACLs needed for connectors to create their own topics (with CREATE ACL operation on the CLUSTER resource). You can override this behaviour by setting the config below to false. And instead create the needed topics with Julie Ops.

Property: topology.connector.allow.topic.create Default value: true

An example configuration will look like this:

topology.connector.allow.topic.create=false

Retrieve topic management state from local controlled view

By default since it’s creation Julie Ops has been retrieving the state of topics from the target cluster, this means pulling the actual view directly from there (AK cluster) using AdminClient. To disable this it can be done below.

If you want to manage the current view of topics from the own Julie Ops cluster state subsystem, you should use this property.

Property: topology.state.topics.cluster.enabled Default value: true

This property is for the time being true as default (backwards compatible).

An example to use local topic management state will look like this:

topology.state.topics.cluster.enabled=false

Retrieve management state from local controlled view

Julie Ops for everything apart from topics uses a local state, so that Julie Ops’s uses the actual state not its internal state this means pulling the actual view directly, we can enable this for everything, topics, acls, service accounts etc. Note this flag supercedes the topology.state.topics.cluster.enabled.

If you want to manage the current view of everything from the own Julie Ops cluster state subsystem, you should use this property.

Property: topology.state.cluster.enabled Default value: true

This property is for the time being false as default (backwards compatible).

An example to use actual view management state will look like this:

topology.state.cluster.enabled=false

Control allowed Topics to be managed by Julie Ops

This property is used to control which Topics are allowed to be managed by Julie Ops, this variable contains a list of allowed prefixes.

Property: topology.topic.managed.prefixes Default value: “[]”

An example configuration might look like this:

topology.topic.managed.prefixes.0=User:AService
topology.topic.managed.prefixes.1=User:BService

If this prefix list is used, only topics that match the prefix will be ever processed, anything else will be ignored. This is useful in a shared cluster, to avoid Julie Ops removing/accidentally managing topics managed by other teams with seperate pipelines.

Control allowed Service accounts to be managed by Julie Ops

This property is used to control which Service Accounts are allowed to be managed by Julie Ops, this variable contains a list of allowed prefixes.

Property: topology.service.accounts.managed.prefixes Default value: “[]”

An example configuration might look like this:

topology.service.accounts.managed.prefixes.0=User:AService
topology.service.accounts.managed.prefixes.1=User:BService

If this prefix list is used, only service accounts that match the prefix will be ever processed, anything else will be ignored. This is useful in a shared cluster, to avoid Julie Ops removing/accidentally managing service accounts managed by other teams with seperate pipelines.

Control allowed Group to be managed by Julie Ops

Note, currently Julie Ops just manages Group ACLS.

This property is used to control which Group prefixes are allowed to be managed by Julie Ops, this variable contains a list of allowed prefixes.

Property: topology.group.managed.prefixes Default value: “[]”

An example configuration might look like this:

topology.group.managed.prefixes.0=NameSpaceA
topology.group.managed.prefixes.1=NameSpaceB

If this prefix list is used, only groups that match the prefix will be ever processed, if wildcard it will be managed if the service account is managed by Julie Ops, anything else will be ignored. This is useful in a shared cluster, to avoid Julie Ops removing/accidentally managing group acls by other teams with seperate pipelines.