Using below configuration I am able to connect samza to kafka-broker
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
systems.kafka.samza.msg.serde=json
systems.kafka.consumer.zookeeper.connect=localhost:2181/
systems.kafka.producer.bootstrap.servers=localhost:9092
But I'm have some doubts regarding SystemFactory class. How to write our own systemfactory class? and what is the purpose of SystemFactoryClass? please give me some idea
You can write your own system factory class by extending the SystemFactory interface and implementing its three abstract functions, getConsumer, getProducer, and getAdmin. In each one of the functions, take getConsumer as an example, you want to create a system customer, an instance of another customized class extending SystemConsumer and defining how the system should consume. By doing so, your Samza job would know how to get the admin/consumer/producer of the system when needed.
Example (in Scala):
class YourSystemFactory extends SystemFactory {
override def getConsumer(systemName: String, config: Config, registry: MetricsRegistry): SystemConsumer = {
new YourSystemConsumer(
getAdmin(systemName, config).asInstanceOf[YourSystemAdmin],
config.get("someParam"))
}
override def getAdmin(systemName: String, config: Config): SystemAdmin = {
new YourSystemAdmin(
config.get("someParam"),
config.get("someOtherParam"))
)
}
override def getProducer(systemName: String, config: Config, registry: MetricsRegistry): SystemProducer = {
new YourSystemProducer(
getAdmin(systemName, config).asInstanceOf[YourSystemAdmin],
config.get("someParam"))
}
}
In your config:
# Your system params
systems.your.samza.factory=your.package.YourSystemFactory
systems.your.consumer.param=value
systems.your.producer.param=value
You don't need implement your KafkaSystemFactory. You have just implement StreamTask
Example :
public class MyTaskClass implements StreamTask {
public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) {
// process message
}
}
Config :
# This is the class above, which Samza will instantiate when the job is run
task.class=com.example.samza.MyTaskClass
# Define a system called "kafka" (you can give it any name, and you can define
# multiple systems if you want to process messages from different sources)
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
# The job consumes a topic called "PageViewEvent" from the "kafka" system
task.inputs=kafka.PageViewEvent
# Define a serializer/deserializer called "json" which parses JSON messages
serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory
# Use the "json" serializer for messages in the "PageViewEvent" topic
systems.kafka.streams.PageViewEvent.samza.msg.serde=json
For more info : Documentation
Related
I have many endpoints in my app:
/Route1
/Route2
...
/Route99
In a number of these routes, there is some common functionality such as getting specific data from one source such as a local file, or another resource such as a No SQL database or external HTTP endpoint. My problem is that these services need to have a service dependency themselves, and I am not sure that how I have currently done it is the best way to do it in NestJS.
Route1Service - Read a file of data, and return it. This uses the FileSystemService() to wrap all the error handling, different data types, path checking etc., of the NodeJS fs module. The Route1Service then returns this to the Route1Controller
#Injectable()
export class Route1Service {
private FS_:FileSystemService; // defined here instead of constructor, as I do not know how to set it in the constructor via NestJS, or if this is even the best way.
// constructor(private FS_: FileSystemService) { }
// Since I do not set it in the constructor
public DataServiceDI(FsService:FileSystemService):void {
this.FS_ = FsService;
}
public GetData(): string {
const Data:string = this.FS_.ReadLocalFile('a.txt');
return Data;
}
}
Route99Service might do the same thing, but with a different file (b.txt)
#Injectable()
export class Route99Service {
private FS_:FileSystemService;
public DataServiceDI(FsService:FileSystemService):void {
this.FS_ = FsService;
}
public GetData(): string {
const Data:string = this.FS_.ReadLocalFile('b.txt');
return Data;
}
}
This is a contrived example to illustrate my issue. Obviously a basic RouteService could be used, and pass the file name, but I am trying to illustrate the dependent service. I do not know how to define the module(s) to use this dependent service or if I should be doing it this way.
What I have been doing for my definition:
#Module({
controllers: [Route1Controller],
providers: [Route1Service, FileSystemService],
})
export class Route1Module {}
The controller than has the constructor with both Services:
#Controller('route1')
export class Route1Controller
constructor(
private Route1_: Route1Service,
private FsSystem_: FileSystemService
) { }
Now that my controller has the FsSystem service as a separate entity, I need to add a method on my Route1Service, DataServiceDI(), to allow me to pass the FileSystemService as a reference. Then my service can use this service to access the file system.
My question comes down to, is this the best practice for this sort of thing? Ultimately, in my code, these services (FileSystemService, NoSqlService) extend a common service type, so that all my services can have this DataServiceDI() in then (they extend a base service with this definition).
Is this the best approach for longer term maintainability? Is there an easier way to simply inject the proper service into my Route1Service so it is injected by NestJS, and I do not have to do the DI each time?
The current method works for me to be able to simply test the service, since I can easily mock the FileSystemServie, NoSqlService, etc., and then inject the mock.
I'm currently working on a project created with api-platform and using Symfony4.
My goal is to be able to use a TokenStorage inside of an EntityListener in order to know the identity of my user.
Unfortunately, I can't find a way to pass a TokenStorageInterface as an argument.
And the autowiring system doesn't seem to work.
services.yaml :
# https://symfony.com/doc/current/best_practices/configuration.html#application-related-configuration
parameters:
locale: 'en'
services:
# default configuration for services in *this* file
_defaults:
autowire: true # Automatically injects dependencies in your services.
autoconfigure: true # Automatically registers your services as commands, event subscribers, etc.
public: false # Allows optimizing the container by removing unused services; this also means
# fetching services directly from the container via $container->get() won't work.
# The best practice is to be explicit about your dependencies anyway.
# makes classes in src/ available to be used as services
# this creates a service per class whose id is the fully-qualified class name
App\:
resource: '../src/*'
exclude: '../src/{Entity,Migrations,Tests}'
# controllers are imported separately to make sure services can be injected
# as action arguments even if you don't extend any base controller class
App\Controller\:
resource: '../src/Controller'
tags: ['controller.service_arguments']
# add more service definitions when explicit configuration is needed
# please note that last definitions always *replace* previous ones
App\EntityListener\KpiDateListener:
public: true,
arguments:
$tokenStorage: '#security.token_storage'
KpiDateListener.php :
<?php
namespace App\EntityListener;
use ApiPlatform\Core\Tests\Fixtures\TestBundle\Manager\UserManager;
use App\Entity\KpiDate;
use App\Entity\StateDate;
use App\Security\User\UserProvider;
use Doctrine\ORM\Event\LifecycleEventArgs;
use Doctrine\ORM\OptimisticLockException;
use Doctrine\ORM\ORMException;
use Psr\Log\LoggerInterface;
use Symfony\Component\Security\Core\Authentication\AuthenticationProviderManager;
use Symfony\Component\Security\Core\Authentication\Token\Storage\TokenStorage;
use Symfony\Component\Security\Core\User\UserInterface;
use Symfony\Component\Security\Core\Authentication\Token\Storage\TokenStorageInterface;
class KpiDateListener
{
private $tokenStorage;
public function __construct(TokenStorageInterface $tokenStorage)
{
$this->tokenStorage = $tokenStorage;
}
public function postPersist(KpiDate $kpiDate, LifecycleEventArgs $args)
{
//Here is my function
}
}
I always have the same error :
Too few arguments to function App\EntityListener\KpiDateListener::__construct(), 0 passed in /srv/api/vendor/doctrine/doctrine-bundle/Mapping/ContainerAwareEntityListenerResolver.php on line 76 and exactly 1 expected
Is there something wrong or missing ?
Thank you
The error message would indicate that autowiring is not enabled for your listener. Autowiring works for services. If your entity listener is not registered as a service, the DI component cannot autowire it, as it does not know about it (and then, instantiating the class will be performed by Doctrine itself).
Try changing to:
App\EntityListener\KpiDateListener:
autowire: true
public: true,
arguments:
$tokenStorage: '#security.token_storage'
tags:
- { name: doctrine.orm.entity_listener }
I'm working with Guice and have one design question. My App consists of few module:
myapp-persistence (JPA Entities, DAO, other DB related stuff)
myapp-backend (Some background daemons, they use myapp-persistence )
myapp-rest (REST app that depends on myapp-persistence)
myapp-persistence must have singleton HibernateSessionFactory. It's by Hibernate design.
No problem I can solve it with Guice:
class MyAppPersistenceModule extends AbstractModule {
override def configure(): Unit = {
bind(classOf[SomeStuff])
bind(classOf[ClientDao])
bind(classOf[CustomerDao])
bind(classOf[SessionFactory]).toProvider(classOf[HibernateSessionFactoryProvider]).asEagerSingleton()
}
#Provides
def provideDatabaseConnectionConfiguration: DatabaseConnectionConfiguration = {
DatabaseConnectionConfiguration.fromSysEnv
}
}
The problem with passing DatabaseConnectionConfiguration to that singleton. myapp-persistence module doesn't really care how to get that config. Right now it's taken from sys variables.
myapp-rest is play-app and it wants to read conf from application.conf and inject it into other components using Guice.
myapp-backend does more or less the same.
Right now I'm locked myself with
#Provides
def provideDatabaseConnectionConfiguration: DatabaseConnectionConfiguration = {
DatabaseConnectionConfiguration.fromSysEnv
}
And I don't understand how to make it flexible and configurable for myapp-rest and myapp-backend.
UPD
According to answer, I did it this way:
Defined trait
trait DbConfProvider {
def dbConf: DbConf
}
Singleton factory now depends on provider:
class HibernateSessionFactoryProvider #Inject()(dbConfProvider: DbConfProvider) extends Provider[SessionFactory] {
}
myapp-persistence module exposes public guice module with all piblic persistence module DAO.
myapp-persistence has module used only for testing purposes. myapp-persistence Injector load module described below:
class MyAppPersistenceDbConfModule extends AbstractModule {
override def configure(): Unit = {
bind(classOf[DbConfProvider]).to(classOf[DbConfSysEnvProvider])
}
}
DbConfSysEnvProvider reads DB connection settings from sys env. Non production use case.
Play app has it's own conf mechanism. I've added my custom module to app conf:
# play-specific config
play.modules.enabled += "common.components.MyAppPersistenceDbConfModule"
# public components from myapp-persistence module.
play.modules.enabled += "com.myapp.persistence.connection.PersistenceModule"
And my configuration service:
#Singleton
class ConfigurationService #Inject()(configuration: Configuration) extends DbConfProvider {
...}
I am not an expert on Play-specific setup, but generally this kind of design problem is solved in one of the following ways:
No default. Remove the binding of DatabaseConnectionConfiguration from the upstream module (myapp-persistence), and define it in each downstream module (myapp-backend, myapp-rest) as appropriate.
Default with override. Keep the default binding of DatabaseConnectionConfiguration like you did, implementing the most common configuration strategy there. Override it in downstream modules using Guice Modules.override(..) API when needed.
Implement a unified configuration mechanism across the modules, that does not depend on particular frameworks used. (E.g. Bootique, which is built on Guice ... Haven't used it with Play though).
I personally prefer the approach #3, but in the absence of something like Bootique, #2 is a good substitute.
I have FlatSpec test classes which need to make use of a REST service for some fixture data. When running all the tests in one go I only really want to instantiate the REST client once as it may be quite expensive. How can I go about this and can I also get it to work for running just one test class when I am running in my IDE?
1. Use mocking:
I would advice you to use some kind of mocking when you try to test REST service. You can try for example scala-mock. Creation of mock service isn't time/cpu consuming, so you can create mocks in all your tests and don't need to share them.
Look:
trait MyRestService {
def get(): String
}
class MyOtherService(val myRestService: MyRestService) {
def addParentheses = s"""(${myRestService.getClass()})"""
}
import org.scalamock.scalatest.MockFactory
class MySpec extends FreeSpec with MockFactory {
"test1 " in {
// create mock rest service and define it's behaviour
val myRestService = mock[MyRestService]
val myOtherService = new MyOtherService(myRestService)
inAnyOrder {
(myRestService.get _).expects().returning("ABC")
}
//now it's ready, you can use it
assert(myOtherService.addParentheses === "(ABC)")
}
}
2. Or use Sharing fixtures:
If you still want to use real implementation of you REST service and create only one instance and then share it with some test condider using:
get-fixture methods => Use it when you need the same mutable fixture objects in multiple tests, and don't need to clean up after.
withFixture(OneArgTest) => Use when all or most tests need the same fixtures that must be cleaned up afterwords.
Refer to http://www.scalatest.org/user_guide/sharing_fixtures#loanFixtureMethods for more details and code examples.
If you want to share the same fixture against multiple Suites use org.scalatest.Suites and #DoNotDiscover annotation (these requires at least scalatest-2.0.RC1)
Pawel's last comment fits well.
It was easier by inheriting from Suite with BeforaAndAfterAll instead of Suites.
import com.typesafe.config.ConfigFactory
import com.google.inject.Guice
import org.scalatest.{BeforeAndAfterAll, Suite}
import net.codingwell.scalaguice.InjectorExtensions.ScalaInjector
class EndToEndSuite extends Suite with BeforeAndAfterAll {
private val injector = {
val config = ConfigFactory.load
val module = new AppModule(config) // your module here
new ScalaInjector(Guice.createInjector(module))
}
override def afterAll {
// your shutdown if needed
}
override val nestedSuites = collection.immutable.IndexedSeq(
injector.instance[YourTest1],
injector.instance[YourTest2] //...
)
}
In symfony 1.4 you could define a custom route class, where you override the generation of url with custom logic, for example:
custom:
class: sfDoctrineRouteCollection
options:
model: Custom
prefix_path: /custom/category/:category_id
column: id
route_class: CustomDoctrineRoute
class CustomDoctrineRoute extends sfDoctrineRoute
{
public function generate($params, $context = array(), $absolute = false)
{
if (!isset($params['category_id'])) {
$params['category_id'] = sfContext::getInstance()->getRequest()->getParameter('category_id');
}
return parent::generate($params, $context, $absolute);
}
}
This allows to write url_for('custom_show', array('id'=> $object['id'])) and not bother about context dependent parameters (category_id).
How do you approach this is symfony2?
I can think of 2 approaches to this. The first, and simplest, is to extend the Router class with your own and tell symfony to use your class in your parameters.yml or config.yml:
parameters:
router.class: Company\CoreBundle\Routing\MyCustomRouter
There's a more powerful (read: complicated) solution which allows you to define more dependencies on your router class by overriding or extending the whole router service. There is a bundle that does this called BeSimpleI18nRoutingBundle which you can look at to see how it's done.
Specifically, notice the CompilerPass where they replace the default router service with their own. You then have to implement the RouterInterface in your own router class. In this particular bundle they inject the original default router (after having renamed it in the compiler pass).