4

I am trying to setup and understand Cygnus. But I am facing issue during installation.

I followed below given steps.

  1. Install Cygnus using Docker (docker run -d -p 5050:5050 -p 8081:8081 fiware/cygnus-common)

  2. Executed version command (curl http://172.17.0.2:8081/v1/version) which gave following response {"success":"true","version":"1.8.0_SNAPSHOT.39b2aa4789c61fa92fe6edc905410f1ddeb33490"}

  3. Login into Cygnus container using command docker exec -it /bin/bash

  4. Created new file named “agent_mysql.conf” in “/opt/apache-flume/conf/” folder.

    Configuration details given below

.

cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = mysql-sink
cygnus-ngsi.channels = mysql-channel

cygnus-ngsi.sources.http-source.channels = mysql-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = def_serv
cygnus-ngsi.sources.http-source.handler.default_service_path = def_servpath
cygnus-ngsi.sources.http-source.handler.events_ttl = 2
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /Applications/apache-flume-1.4.0-bin/conf/grouping_rules.conf

# =============================================
# mysql-channel configuration
# channel type (must not be changed)
cygnus-ngsi.channels.mysql-channel.type = memory
# capacity of the channel
cygnus-ngsi.channels.mysql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnus-ngsi.channels.mysql-channel.transactionCapacity = 100



# channel name from where to read notification events
cygnus-ngsi.sinks.mysql-sink.channel = mysql-channel
# sink class, must not be changed
cygnus-ngsi.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIMySQLSink
#com.telefonica.iot.cygnus.sinks.OrionMySQLSink
# the FQDN/IP address where the MySQL server runs
cygnus-ngsi.sinks.mysql-sink.mysql_host = localhost
# the port where the MySQL server listes for incomming connections
cygnus-ngsi.sinks.mysql-sink.mysql_port = 3306
# a valid user in the MySQL server
cygnus-ngsi.sinks.mysql-sink.mysql_username = root
# password for the user above
cygnus-ngsi.sinks.mysql-sink.mysql_password = <myPassword>
# how the attributes are stored, either per row either per column (row, column)
cygnus-ngsi.sinks.mysql-sink.attr_persistence = row
  1. Changed "cygnus-entrypoint.sh" file in / (root) folder and added following command by removing existing one. ${FLUME_HOME}/bin/cygnus-flume-ng agent --conf ${CYGNUS_CONF_PATH} -f ${CYGNUS_CONF_PATH}/agent_mysql.conf -n cygnus-ngsi -p ${CYGNUS_API_PORT} -Dflume.root.logger=${CYGNUS_LOG_LEVEL},${CYGNUS_LOG_APPENDER} -Dfile.encoding=UTF-8
  2. Exited Docker container and came back to Ubuntu.
  3. Stop and restart Docker container.
  4. And I am getting following errors in logs

Please check and let me know what am I doing wrong? Appreciate your help.

LOGS

n$AgentConfiguration[1016] : Processing:mysql-sink
time=2018-04-30T14:24:00.807Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=validateConfiguration | msg=org.apache.flume.conf.FlumeConfiguration[140] : Post-validation flume configuration contains configuration for agents: [cygnus-ngsi]
time=2018-04-30T14:24:00.808Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=loadChannels | msg=org.apache.flume.node.AbstractConfigurationProvider[150] : Creating channels
time=2018-04-30T14:24:00.816Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=create | msg=org.apache.flume.channel.DefaultChannelFactory[40] : Creating instance of channel mysql-channel type memory
time=2018-04-30T14:24:00.825Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=loadChannels | msg=org.apache.flume.node.AbstractConfigurationProvider[205] : Created channel mysql-channel
time=2018-04-30T14:24:00.832Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=create | msg=org.apache.flume.source.DefaultSourceFactory[39] : Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
time=2018-04-30T14:24:00.836Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=configure | msg=org.apache.flume.source.http.HTTPSource[113] : Error while configuring HTTPSource. Exception follows.
java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.NGSIRestHandler
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:102)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
time=2018-04-30T14:24:00.840Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=loadSources | msg=org.apache.flume.node.AbstractConfigurationProvider[366] : Source http-source has been removed due to an error during configuration
java.lang.RuntimeException: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.NGSIRestHandler
        at com.google.common.base.Throwables.propagate(Throwables.java:156)
        at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:114)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.NGSIRestHandler
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:102)
        ... 11 more
time=2018-04-30T14:24:00.841Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=create | msg=org.apache.flume.sink.DefaultSinkFactory[40] : Creating instance of sink: mysql-sink, type: com.telefonica.iot.cygnus.sinks.NGSIMySQLSink
time=2018-04-30T14:24:00.842Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=run | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[142] : Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load sink type: com.telefonica.iot.cygnus.sinks.NGSIMySQLSink, class: com.telefonica.iot.cygnus.sinks.NGSIMySQLSink
        at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
        at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.sinks.NGSIMySQLSink
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
        ... 11 more
Krishan Babbar
  • 766
  • 1
  • 7
  • 17

2 Answers2

0

The simplest case is installing cygnus connecting MYSQL this way, using the "root" user to connect to cygnus.

docker run -d --name cygnus_container_name --link mysql_showcases \
  -p 8081:8081 -p 5050:5050 \
  -e CYGNUS_MYSQL_HOST=mysql_host -e CYGNUS_MYSQL_PORT=3306 \
  -e CYGNUS_MYSQL_USER=root -e CYGNUS_MYSQL_PASS=root_password \
  fiware/cygnus-ngsi

If you decide not to use root user to connect mysql, you'll need to change your user and password and create the database manually and granting the permissions to your user, since cygnus won't be able to create a database with a different user.

jicarretero
  • 331
  • 1
  • 2
  • Thanks @jicarretero, I ran following commands 1. docker run --name=mysqlser -d mysql/mysql-server 2. docker run -d --name cygnus-mysql1 --link mysqlser -p 8081:8081 -p 5050:5050 -e CYGNUS_MYSQL_HOST=172.17.0.2 -e CYGNUS_MYSQL_PORT=3306 -e CYGNUS_MYSQL_USER=root -e CYGNUS_MYSQL_PASS= fiware/cygnus-ngsi I logged into MySQL Container but did not see any DB. I could see only following DBs. docker exec -it mysqlser mysql -u root -p mysql> show databases; | information_schema | | mysql | | performance_schema | | sys What is expected? – Krishan Babbar May 04 '18 at 09:33
0

Finally I am able to run Cygnus with MySQL agent. I am using Ubuntu. (Linux ubuntucustomfiware 4.4.0-119-generic #143-Ubuntu SMP Mon Apr 2 16:08:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux)

I followed below steps.

  1. Used MySQL installed in main Ubuntu instance instead of Docker container.

Modified /etc/mysql/mysql.conf.d/mysqld.cnf and changed

from

bind-address = 127.0.0.1

to

bind-address = *

  1. Login into DB and grant all privileges to root user, so that it can connect from any host.

mysql -u root -p

GRANT ALL PRIVILEGES ON . TO 'root'@'%' IDENTIFIED BY 'MyPassword';

FLUSH PRIVILEGES;

exit;

Restart MySQL server

service mysql restart

  1. Run Cygnus-ngsi

docker run -d --name cygnus -p 8081:8081 -p 5050:5050 -e CYGNUS_MYSQL_HOST=PublicIPOfMySQLServer -e CYGNUS_MYSQL_PORT=3306 -e CYGNUS_MYSQL_USER=root -e CYGNUS_MYSQL_PASS=MyPassword -e CYGNUS_LOG_LEVEL='DEBUG' fiware/cygnus-ngsi

  1. Modified Agent file and keep only mysql-sink. After below changes, stop/start cygnus docker container.

docker exec -it cygnus /bin/bash

vi /opt/apache-flume/conf/agent.conf

cygnus-ngsi.sinks = mysql-sink

cygnus-ngsi.channels = mysql-channel

exit;

docker stop cygnus

docker start cygnus

  1. Now publish MQTT data to modify my entity and it inserted 4 rows (one row for each attribute) into MySQL DB

mosquitto_pub -h PublicIPOfMySQLServer -u UserName -P Password -t /swm-reader-service1/reader-device-id1/attrs -m '{"tn": "9888", "pn": "878787", "ri": "888888", "tdt":"Monday, May 10, 2018 03:16 AM"}'

Thanks for all your support.

Regards,

Krishan

Community
  • 1
  • 1
Krishan Babbar
  • 766
  • 1
  • 7
  • 17