■ドキュメント

RHEL7 IDENTITY MANAGEMENT

■参考

FreeIPAの紹介(スライド)

 

■インストール要件

サーバーインストールの前提条件

 

■インストール

上記のドキュメントの①のパート II.IDENTITY MANAGEMENT のインストール参照

 

●検証環境の構成・方式

・構成

※下記のような構成でIdMサーバ1台IdMクライアント1台ずつとする。

 

         ホスト名                    IPアドレス         メモリ   CPU 

IdMサーバ  centos7.dagyah.com      192.168.19.200   4G      2vcpu

IdMクライアント centos7copy.dagyah.com 192.168.19.201    4G      2vcpu

 

※IdMサーバとIdMクライアントはいずれも下記の通り

 

OS             CentOS7.4

IdMバージョン           4.6.6-11.el7        

・方式

 - 統合DNSを使用

 - CAあり

 - DNS フォワーダーなし

 

●準備

・チャラい準備

[centos7]$ setenforce 0
[centos7]$ getenforce

Permissive
[centos7]$ iptables -F

 

●インストール(対話式)

[centos7]$ yum install ipa-server ipa-server-dns

----(略)----

Installed:
 ipa-server.x86_64 0:4.6.6-11.el7.centos                    ipa-server-dns.noarch 0:4.6.6-11.el7.centos

DependencyInstalled:
 389-ds-base.x86_640:1.3.10.1-14.el7_8
 389-ds-base-libs.x86_640:1.3.10.1-14.el7_8
 antlr-tool.noarch 0:2.7.7-30.el7
 apache-commons-cli.noarch0:1.2-13.el7
 apache-commons-codec.noarch0:1.8-7.el7
 apache-commons-collections.noarch0:3.2.1-22.el7_2
 apache-commons-daemon.x86_640:1.0.13-7.el7
 apache-commons-dbcp.noarch0:1.4-17.el7
 apache-commons-io.noarch1:2.4-12.el7
 apache-commons-lang.noarch0:2.6-15.el7
 apache-commons-logging.noarch0:1.1.2-7.el7
 apache-commons-pool.noarch 0:1.6-9.el7
 apr.x86_640:1.4.8-5.el7
 apr-util.x86_64 0:1.5.2-6.el7
 apr-util-openssl.x86_640:1.5.2-6.el7
 args4j.noarch0:2.0.16-13.el7
 avalon-framework.noarch0:4.3-10.el7
 avalon-logkit.noarch0:2.1-14.el7
 bcel.noarch0:5.2-18.el7
 bea-stax.noarch 0:1.2.0-9.el7
 bea-stax-api.noarch0:1.2.0-9.el7
 bind.x86_6432:9.11.4-16.P2.el7_8.6
 bind-dyndb-ldap.x86_640:11.1-7.el7
 bind-export-libs.x86_6432:9.11.4-16.P2.el7_8.6
 bind-pkcs11.x86_6432:9.11.4-16.P2.el7_8.6
 bind-pkcs11-libs.x86_64 32:9.11.4-16.P2.el7_8.6
 bind-pkcs11-utils.x86_6432:9.11.4-16.P2.el7_8.6
 cal10n.noarch0:0.7.7-4.el7
 codemodel.noarch0:2.6-9.el7
 custodia.noarch0:0.3.1-4.el7
 dom4j.noarch0:1.6.1-20.el7
 easymock2.noarch0:2.5.2-12.el7
 ecj.x86_64 1:4.5.2-3.el7
 fontawesome-fonts.noarch0:4.1.0-2.el7
 geronimo-jms.noarch0:1.1.1-19.el7
 geronimo-jta.noarch0:1.1.1-17.el7
 glassfish-dtd-parser.noarch0:1.2-0.8.20120120svn.el7
 glassfish-fastinfoset.noarch0:1.2.12-9.el7
 glassfish-jaxb.noarch 0:2.2.5-6.el7
 glassfish-jaxb-api.noarch0:2.2.7-4.el7
 hamcrest.noarch0:1.3-6.el7
 hsqldb.noarch1:1.8.1.3-14.el7
 httpcomponents-client.noarch0:4.2.5-5.el7_0
 httpcomponents-core.noarch0:4.2.4-6.el7
 httpd.x86_640:2.4.6-93.el7.centos
 httpd-tools.x86_640:2.4.6-93.el7.centos
 ipa-server-common.noarch0:4.6.6-11.el7.centos
 isorelax.noarch1:0-0.15.release20050331.el7
 istack-commons.noarch0:2.17-4.el7
 jackson.noarch0:1.9.4-7.el7
 jakarta-commons-httpclient.noarch1:3.1-16.el7_0
 jakarta-oro.noarch0:2.0.8-16.el7
 javamail.noarch0:1.4.6-8.el7
 javassist.noarch0:3.16.1-10.el7
 jaxen.noarch0:1.1.3-11.el7
 jboss-annotations-1.1-api.noarch0:1.0.1-0.6.20120212git76e1a2.el7
 jdom.noarch0:1.1.3-6.el7
 jing.noarch0:20091111-14.el7
 joda-convert.noarch0:1.3-5.el7
 joda-time.noarch 0:2.2-3.tzdata2013c.el7
 jsr-311.noarch0:1.1.1-6.el7
 jss.x86_640:4.4.7-2.el7
 junit.noarch0:4.11-8.el7
 jvnet-parent.noarch0:4-2.el7
 krb5-pkinit.x86_640:1.15.1-46.el7
 krb5-server.x86_640:1.15.1-46.el7
 ldapjdk.noarch0:4.19-5.el7
 ldns.x86_640:1.6.16-10.el7
 libitm.x86_640:4.8.5-39.el7
 log4j.noarch0:1.2.17-16.el7_4
 mailcap.noarch 0:2.1.41-2.el7
 mod_auth_gssapi.x86_640:1.5.1-5.el7
 mod_lookup_identity.x86_640:1.0.0-1.el7
 mod_nss.x86_640:1.0.14-12.el7
 mod_session.x86_640:2.4.6-93.el7.centos
 mod_wsgi.x86_640:3.4-18.el7
 msv-msv.noarch1:2013.5.1-7.el7
 msv-xsdlib.noarch 1:2013.5.1-7.el7
 nuxwdog.x86_640:1.0.5-1.el7
 nuxwdog-client-java.x86_640:1.0.5-1.el7
 objectweb-asm.noarch0:3.3.1-9.el7
 opencryptoki.x86_640:3.12.1-2.el7
 opencryptoki-libs.x86_640:3.12.1-2.el7
 opencryptoki-swtok.x86_640:3.12.1-2.el7
 opendnssec.x86_640:1.4.7-4.el7
 openldap-clients.x86_640:2.4.44-21.el7_6
 perl-Archive-Tar.noarch0:1.92-3.el7
 perl-Compress-Raw-Bzip2.x86_640:2.061-3.el7
 perl-Compress-Raw-Zlib.x86_641:2.061-4.el7
 perl-DB_File.x86_64 0:1.830-6.el7
 perl-Data-Dumper.x86_640:2.145-3.el7
 perl-IO-Compress.noarch0:2.061-2.el7
 perl-IO-Zlib.noarch 1:1.10-295.el7
 perl-Mozilla-LDAP.x86_640:1.5.3-12.el7
 perl-NetAddr-IP.x86_640:4.069-3.el7
 perl-Package-Constants.noarch1:0.02-295.el7
 pki-base.noarch0:10.5.17-6.el7
 pki-base-java.noarch0:10.5.17-6.el7
 pki-ca.noarch0:10.5.17-6.el7
 pki-kra.noarch0:10.5.17-6.el7
 pki-server.noarch 0:10.5.17-6.el7
 pki-tools.x86_640:10.5.17-6.el7
 python-custodia.noarch0:0.3.1-4.el7
 python-kdcproxy.noarch 0:0.3.2-3.el7
 python2-ipaserver.noarch0:4.6.6-11.el7.centos
 qdox.noarch0:1.12.1-10.el7
 regexp.noarch0:1.5-13.el7
 relaxngDatatype.noarch0:1.0-11.el7
 resteasy-base-atom-provider.noarch0:3.0.6-4.el7
 resteasy-base-client.noarch 0:3.0.6-4.el7
 resteasy-base-jackson-provider.noarch0:3.0.6-4.el7
 resteasy-base-jaxb-provider.noarch0:3.0.6-4.el7
 resteasy-base-jaxrs.noarch 0:3.0.6-4.el7
 resteasy-base-jaxrs-api.noarch0:3.0.6-4.el7
 rngom.noarch0:201103-0.8.20120119svn.el7
 scannotation.noarch0:1.0.3-0.7.r12.el7
 slapi-nis.x86_640:0.56.0-13.el7
 slf4j.noarch0:1.7.4-4.el7_4
 softhsm.x86_640:2.1.0-2.el7
 sssd-dbus.x86_640:1.16.4-37.el7_8.4
 stax-ex.noarch0:1.7.1-6.el7
 stax2-api.noarch0:3.1.1-10.el7
 svrcore.x86_64 0:4.1.3-2.el7
 tomcat.noarch0:7.0.76-12.el7_8
 tomcat-el-2.2-api.noarch0:7.0.76-12.el7_8
 tomcat-jsp-2.2-api.noarch0:7.0.76-12.el7_8
 tomcat-lib.noarch0:7.0.76-12.el7_8
 tomcat-servlet-3.0-api.noarch0:7.0.76-12.el7_8
 tomcatjss.noarch0:7.2.5-1.el7
 txw2.noarch0:20110809-8.el7
 velocity.noarch0:1.7-10.el7
 ws-jaxme.noarch0:0.5.2-10.el7
 xalan-j2.noarch0:2.7.1-23.el7
 xerces-j2.noarch 0:2.11.0-17.el7_0
 xml-commons-apis.noarch0:1.4.01-16.el7
 xml-commons-resolver.noarch0:1.2-15.el7
 xpp3.noarch0:1.1.3.8-11.el7
 xsom.noarch0:0-10.20110809svn.el7
 DependencyUpdated:
 bind-libs.x86_6432:9.11.4-16.P2.el7_8.6
 bind-libs-lite.x86_6432:9.11.4-16.P2.el7_8.6
 bind-license.noarch32:9.11.4-16.P2.el7_8.6
 bind-utils.x86_6432:9.11.4-16.P2.el7_8.6
 certmonger.x86_640:0.78.4-12.el7
 dbus.x86_641:1.10.24-14.el7_8
 dbus-libs.x86_641:1.10.24-14.el7_8
 dbus-x11.x86_641:1.10.24-14.el7_8
 dhclient.x86_6412:4.2.5-79.el7.centos
 dhcp-common.x86_6412:4.2.5-79.el7.centos
 dhcp-libs.x86_6412:4.2.5-79.el7.centos
 ipa-client.x86_640:4.6.6-11.el7.centos
 ipa-client-common.noarch0:4.6.6-11.el7.centos
 ipa-common.noarch0:4.6.6-11.el7.centos
 krb5-libs.x86_640:1.15.1-46.el7
 krb5-workstation.x86_640:1.15.1-46.el7
 libipa_hbac.x86_640:1.16.4-37.el7_8.4
 libkadm5.x86_64 0:1.15.1-46.el7
 libldb.x86_640:1.5.4-1.el7
 libselinux.x86_640:2.5-15.el7
 libselinux-python.x86_640:2.5-15.el7
 libselinux-utils.x86_640:2.5-15.el7
 libsemanage.x86_640:2.5-14.el7
 libsemanage-python.x86_640:2.5-14.el7
 libsepol.x86_640:2.5-10.el7
 libsss_autofs.x86_640:1.16.4-37.el7_8.4
 libsss_idmap.x86_640:1.16.4-37.el7_8.4
 libsss_nss_idmap.x86_640:1.16.4-37.el7_8.4
 libsss_sudo.x86_640:1.16.4-37.el7_8.4
 libtalloc.x86_64 0:2.1.16-1.el7
 libtdb.x86_640:1.3.18-1.el7
 libtevent.x86_640:0.9.39-1.el7
 nspr.x86_640:4.21.0-1.el7
 nss.x86_640:3.44.0-7.el7_7
 nss-softokn.x86_640:3.44.0-8.el7_7
 nss-softokn-freebl.x86_640:3.44.0-8.el7_7
 nss-sysinit.x86_640:3.44.0-7.el7_7
 nss-tools.x86_640:3.44.0-7.el7_7
 nss-util.x86_640:3.44.0-4.el7_7
 openldap.x86_640:2.4.44-21.el7_6
 policycoreutils.x86_640:2.5-34.el7
 policycoreutils-python.x86_640:2.5-34.el7
 python-libipa_hbac.x86_640:1.16.4-37.el7_8.4
 python-netaddr.noarch0:0.7.5-9.el7
 python-sssdconfig.noarch0:1.16.4-37.el7_8.4
 python2-ipaclient.noarch0:4.6.6-11.el7.centos
 python2-ipalib.noarch0:4.6.6-11.el7.centos
 selinux-policy.noarch0:3.13.1-266.el7_8.1
 selinux-policy-targeted.noarch0:3.13.1-266.el7_8.1
 setools-libs.x86_640:3.3.8-4.el7
 sssd.x86_640:1.16.4-37.el7_8.4
 sssd-ad.x86_640:1.16.4-37.el7_8.4
 sssd-client.x86_640:1.16.4-37.el7_8.4
 sssd-common.x86_64 0:1.16.4-37.el7_8.4
 sssd-common-pac.x86_640:1.16.4-37.el7_8.4
 sssd-ipa.x86_640:1.16.4-37.el7_8.4
 sssd-krb5.x86_640:1.16.4-37.el7_8.4
 sssd-krb5-common.x86_640:1.16.4-37.el7_8.4
 sssd-ldap.x86_640:1.16.4-37.el7_8.4
 sssd-proxy.x86_640:1.16.4-37.el7_8.4

Complete!

[centos7]$ rpm -qa|grep ipa-server
ipa-server-common-4.6.6-11.el7.centos.noarch
ipa-server-4.6.6-11.el7.centos.x86_64
ipa-server-dns-4.6.6-11.el7.centos.noarch

[centos7]$ which ipa-server-install
/usr/sbin/ipa-server-install

[centos7]$ ipa-server-install

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure the KDC to enable PKINIT

To accept the default shown in brackets, press the Enter key.

WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd


Do you want to configure integrated DNS (BIND)? [no]: 
yes

 

Enter the fully qualified domain name of the computer
on which you're setting up server software. Using the form
<hostname>.<domainname>
Example: master.example.com.

Server host name [centos7.4]: 
centos7.dagyah.com

 

Warning: skipping DNS resolution of host centos7.dagyah.com
The domain name has been determined based on the host name.


Please confirm the domain name [dagyah.com]: 
dagyah.com

The kerberos protocol requires a Realm name to be defined.
This is typically the domain name converted to uppercase.

Please provide a realm name [DAGYAH.COM]:
DAGYAH.COM

Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and has full access
to the Directory for system management tasks and will be added to the
instance of directory server created for IPA.
The password must be at least 8 characters long.

Directory Manager password:
Password123

Password (confirm):Password123

The IPA server requires an administrative user, named 'admin'.
This user is a regular system account used for IPA server administration.

IPA admin password:
Password123
Password (confirm):Password123

Checking DNS domain dagyah.com., please wait ...
Enter an additional IP address, or press Enter to skip:
192.168.19.200
Enter an additional IP address, or press Enter to skip:
Do you want to configure DNS forwarders? [yes]:
yes

Following DNS servers are configured in /etc/resolv.conf: 192.168.100.1, 192.168.19.1, 192.168.47.2
Do you want to configure these servers as DNS forwarders? [yes]: 
yes
All DNS servers from /etc/resolv.conf were added. You can enter additional addresses now:
Enter an IP address for a DNS forwarder, or press Enter to skip:

Checking DNS forwarders, please wait ...
DNS server 192.168.100.1: answer to query '. SOA' is missing DNSSEC signatures (no RRSIG data)
Please fix forwarder configuration to enable DNSSEC support.
(For BIND 9 add directive "dnssec-enable yes;" to "options {}")

ipaserver.install.bindinstance: ERROR    DNS server 192.168.19.1: query '. SOA': The DNS operation timed out after 10.0007860661 seconds
ipapython.admintool: ERROR    DNS server 192.168.19.1: query '. SOA': The DNS operation timed out after 10.0007860661 seconds
ipapython.admintool: ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

[centos7]$ cat /var/log/ipaserver-install.log

----(略)----

2020-11-06T03:02:58Z DEBUG Checking DNS server: 192.168.100.1
2020-11-06T03:02:58Z WARNING DNS server 192.168.100.1 does not support DNSSEC: answer to query '. SOA' is missing DNSSEC signatures (no RRSIG data)
2020-11-06T03:02:58Z WARNING Please fix forwarder configuration to enable DNSSEC support.
(For BIND 9 add directive "dnssec-enable yes;" to "options {}")
2020-11-06T03:02:58Z DEBUG Checking DNS server: 192.168.19.1
2020-11-06T03:03:08Z ERROR DNS server 192.168.19.1: query '. SOA': The DNS operation timed out after 10.0007860661 seconds
2020-11-06T03:03:08Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run
    return cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 358, in run
    self.validate()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in validate
    for _nothing in self._validator():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 633, in _configure
    next(validator)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 588, in main
    master_install_check(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 250, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 628, in install_check
    dns.install_check(False, api, False, options, host_name)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dns.py", line 297, in install_check
    and not bindinstance.check_forwarders(options.forwarders):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/bindinstance.py", line 536, in check_forwarders
    raise RuntimeError("DNS server %s: %s" % (forwarder, e))

2020-11-06T03:03:08Z DEBUG The ipa-server-install command failed, exception: RuntimeError: DNS server 192.168.19.1: query '. SOA': The DNS operation timed out after 10.0007860661 seconds
2020-11-06T03:03:08Z ERROR DNS server 192.168.19.1: query '. SOA': The DNS operation timed out after 10.0007860661 seconds
2020-11-06T03:03:08Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

→モバイルルータのDNSforwardwerにSOAレコードをクエリした結果がエラーだった臭い

面倒くさいからサーバのリゾルバで見に行くDNSをgoogleの8.8.8.8にするか

[centos7]$ cat /etc/resolv.conf
# Generated by NetworkManager
search localdomain 4
nameserver 192.168.100.1
nameserver 192.168.47.2

[centos7]$ nmtui
---(編集)---
[centos7]$ systemctl restart NetworkManager
[centos7]$ cat /etc/resolv.conf

# Generated by NetworkManager
search localdomain 4
nameserver 8.8.8.8
nameserver 192.168.47.2

[centos7]$ ifdown ens38
Device 'ens38' successfully disconnected.

[centos7]$ ping 192.168.47.2
PING 192.168.47.2 (192.168.47.2) 56(84) bytes of data.
^C
--- 192.168.47.2 ping statistics ---
4 packets transmitted, 0 

[centos7]$ ping -c 1 www.yahoo.co.jp
PING edge12.g.yimg.jp (183.79.219.252) 56(84) bytes of data.
64 bytes from 183.79.219.252 (183.79.219.252): icmp_seq=1 ttl=47 time=68.8 ms
--- edge12.g.yimg.jp ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 68.875/68.875/68.875/0.000 msreceived, 100% packet loss, time 3001ms

これでもういっかいインストール

→ていうか気づいたけどDNSforwardersの設定要らないわ。「no」にしてもう一回インストール

[centos7]$ ipa-server-install

---(略)---

Do you want to configure DNS forwarders? [yes]: no
No DNS forwarders configured
Do you want to search for missing reverse zones? [yes]:
yes
Do you want to create reverse zone for IP 192.168.19.200 [yes]: yes
Please specify the reverse zone name [19.168.192.in-addr.arpa.]: 19.168.192.in-addr.arpa.
Using reverse zone(s) 19.168.192.in-addr.arpa.

The IPA Master Server will be configured with:
Hostname:       centos7.dagyah.com
IP address(es): 192.168.19.200
Domain name:    dagyah.com
Realm name:     DAGYAH.COM

BIND DNS server will be configured to serve IPA domain with:
Forwarders:       No forwarders
Forward policy:   only
Reverse zone(s):  19.168.192.in-addr.arpa.

Continue to configure the system with these values? [no]:
yes

The following operations may take some minutes to complete.
Please wait until the prompt is returned.
---(略)---
Done configuring certificate server (pki-tomcatd).
Configuring directory server (dirsrv)
  [1/3]: configuring TLS for DS instance
  [2/3]: adding CA certificate entry
  [3/3]: restarting directory server
Done configuring directory server (dirsrv).

ipapython.admintool: ERROR    CA did not start in 300.0s
ipapython.admintool: ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

[centos7]$ cat /var/log/ipaserver-install.log

----(略)----

2020-11-06T03:30:30Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2020-11-06T03:30:30Z DEBUG Waiting for CA to start...
2020-11-06T03:30:31Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run
    return cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 655, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 590, in main
    master_install(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 250, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 847, in install
    ca.install_step_1(False, None, options, custodia=custodia)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 391, in install_step_1
    ca.start('pki-tomcat')
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 464, in start
    self.service.start(instance_name, capture_output=capture_output, wait=wait)
  File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 192, in start
    self.wait_until_running()
  File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 186, in wait_until_running
    raise RuntimeError('CA did not start in %ss' % timeout)

2020-11-06T03:30:31Z DEBUG The ipa-server-install command failed, exception: RuntimeError: CA did not start in 300.0s
2020-11-06T03:30:31Z ERROR CA did not start in 300.0s
2020-11-06T03:30:31Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

→もしかしてシステムのホストネームがFQDNにしてないのでCA作成がこけてる?

[centos7]$ hostnamectl set-hostname centos7.dagyah.com

[centos7]$ systemctl reboot

---(待つ)---

[centos7]$ hostname
centos7.dagyah.com

再インストールする前にリゾルバを元に戻したw

[centos7]$ cat /etc/resolv.conf
# Generated by NetworkManager
search localdomain 4
nameserver 192.168.100.1
nameserver 192.168.47.2

[centos7]$ ipa-server-install
The log file for this installation can be found in /var/log/ipaserver-install.log
ipapython.admintool: ERROR    IPA server is already configured on this system.
If you want to reinstall the IPA server, please uninstall it first using 'ipa-server-install --uninstall'.
ipapython.admintool: ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

→アンインストールしなくちゃ

[centos7]$ ipa-server-install --uninstal

This is a NON REVERSIBLE operation and will delete all data and configuration!
It is highly recommended to take a backup of existing data and configuration using ipa-backup utility before proceeding.
Are you sure you want to continue with the uninstall procedure? [no]:
yes

---(略)---

The ipa-client-install command was successful

もういっかい

[centos7]$ ipa-server-install

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure the KDC to enable PKINIT

To accept the default shown in brackets, press the Enter key.

WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd

Do you want to configure integrated DNS (BIND)? [no]:
yes

Enter the fully qualified domain name of the computer
on which you're setting up server software. Using the form
<hostname>.<domainname>
Example: master.example.com.


Server host name [centos7.dagyah.com]:
centos7.dagyah.com

Warning: skipping DNS resolution of host centos7.dagyah.com
The domain name has been determined based on the host name.

Please confirm the domain name [dagyah.com]:
dagyah.com

The kerberos protocol requires a Realm name to be defined.
This is typically the domain name converted to uppercase.

Please provide a realm name [DAGYAH.COM]:
DAGYAH.COM
Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and has full access
to the Directory for system management tasks and will be added to the
instance of directory server created for IPA.
The password must be at least 8 characters long.

Directory Manager password:
Password123
Password (confirm):Password123

The IPA server requires an administrative user, named 'admin'.
This user is a regular system account used for IPA server administration.

IPA admin password:
Password123
Password (confirm):Password123

Checking DNS domain dagyah.com., please wait ...
Do you want to configure DNS forwarders? [yes]:
no
No DNS forwarders configured
Do you want to search for missing reverse zones? [yes]:
yes
Do you want to create reverse zone for IP 192.168.101.200 [yes]:yes
Please specify the reverse zone name [101.168.192.in-addr.arpa.]: 101.168.192.in-addr.arpa.
Do you want to create reverse zone for IP 192.168.19.200 [yes]: yes
Please specify the reverse zone name [19.168.192.in-addr.arpa.]: 19.168.192.in-addr.arpa.
Do you want to create reverse zone for IP 192.168.47.182 [yes]: no
Do you want to create reverse zone for IP 192.168.122.1 [yes]: no
Using reverse zone(s) 101.168.192.in-addr.arpa., 19.168.192.in-addr.arpa.

The IPA Master Server will be configured with:
Hostname:       centos7.dagyah.com
IP address(es): 192.168.101.200, 192.168.19.200, 192.168.47.182, 192.168.122.1
Domain name:    dagyah.com
Realm name:     DAGYAH.COM

BIND DNS server will be configured to serve IPA domain with:
Forwarders:       No forwarders
Forward policy:   only
Reverse zone(s):  101.168.192.in-addr.arpa., 19.168.192.in-addr.arpa.

Continue to configure the system with these values? [no]:
yes

The following operations may take some minutes to complete.
Please wait until the prompt is returned.

Adding [192.168.101.200 centos7.dagyah.com] to your /etc/hosts file
Adding [192.168.19.200 centos7.dagyah.com] to your /etc/hosts file
Adding [192.168.47.182 centos7.dagyah.com] to your /etc/hosts file
Adding [192.168.122.1 centos7.dagyah.com] to your /etc/hosts file
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/45]: creating directory server instance
  [2/45]: enabling ldapi
  [3/45]: configure autobind for root
  [4/45]: stopping directory server
  [5/45]: updating configuration in dse.ldif
  [6/45]: starting directory server
  [7/45]: adding default schema
  [8/45]: enabling memberof plugin
  [9/45]: enabling winsync plugin
  [10/45]: configure password logging
  [11/45]: configuring replication version plugin
  [12/45]: enabling IPA enrollment plugin
  [13/45]: configuring uniqueness plugin
  [14/45]: configuring uuid plugin
  [15/45]: configuring modrdn plugin
  [16/45]: configuring DNS plugin
  [17/45]: enabling entryUSN plugin
  [18/45]: configuring lockout plugin
  [19/45]: configuring topology plugin
  [20/45]: creating indices
  [21/45]: enabling referential integrity plugin
  [22/45]: configuring certmap.conf
  [23/45]: configure new location for managed entries
  [24/45]: configure dirsrv ccache
  [25/45]: enabling SASL mapping fallback
  [26/45]: restarting directory server
  [27/45]: adding sasl mappings to the directory
  [28/45]: adding default layout
  [29/45]: adding delegation layout
  [30/45]: creating container for managed entries
  [31/45]: configuring user private groups
  [32/45]: configuring netgroups from hostgroups
  [33/45]: creating default Sudo bind user
  [34/45]: creating default Auto Member layout
  [35/45]: adding range check plugin
  [36/45]: creating default HBAC rule allow_all
  [37/45]: adding entries for topology management
  [38/45]: initializing group membership
  [39/45]: adding master entry
  [40/45]: initializing domain level
  [41/45]: configuring Posix uid/gid generation
  [42/45]: adding replication acis
  [43/45]: activating sidgen plugin
  [44/45]: activating extdom plugin
  [45/45]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/10]: adding kerberos container to the directory
  [2/10]: configuring KDC
  [3/10]: initialize kerberos container
  [4/10]: adding default ACIs
  [5/10]: creating a keytab for the directory
  [6/10]: creating a keytab for the machine
  [7/10]: adding the password extension to the directory
  [8/10]: creating anonymous principal
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa-custodia
  [1/5]: Making sure custodia container exists
  [2/5]: Generating ipa-custodia config file
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: reindex attributes
  [3/29]: exporting Dogtag certificate store pin
  [4/29]: stopping certificate server instance to update CS.cfg
  [5/29]: backing up CS.cfg
  [6/29]: disabling nonces
  [7/29]: set up CRL publishing
  [8/29]: enable PKIX certificate path discovery and validation
  [9/29]: starting certificate server instance
  [10/29]: configure certmonger for renewals
  [11/29]: requesting RA certificate from CA
  [12/29]: setting audit signing renewal to 2 years
  [13/29]: restarting certificate server
  [14/29]: publishing the CA certificate
  [15/29]: adding RA agent as a trusted user
  [16/29]: authorizing RA to modify profiles
  [17/29]: authorizing RA to manage lightweight CAs
  [18/29]: Ensure lightweight CAs container exists
  [19/29]: configure certificate renewals
  [20/29]: configure Server-Cert certificate renewal
  [21/29]: Configure HTTP to proxy connections
  [22/29]: restarting certificate server
  [23/29]: updating IPA configuration
  [24/29]: enabling CA instance
  [25/29]: migrating certificate profiles to LDAP
  [26/29]: importing IPA certificate profiles
  [27/29]: adding default CA ACL
  [28/29]: adding 'ipa' CA entry
  [29/29]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).
Configuring directory server (dirsrv)
  [1/3]: configuring TLS for DS instance
  [2/3]: adding CA certificate entry
  [3/3]: restarting directory server
Done configuring directory server (dirsrv).
Configuring ipa-otpd
  [1/2]: starting ipa-otpd
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring the web interface (httpd)
  [1/22]: stopping httpd
  [2/22]: setting mod_nss port to 443
  [3/22]: setting mod_nss cipher suite
  [4/22]: setting mod_nss protocol list to TLSv1.2
  [5/22]: setting mod_nss password file
  [6/22]: enabling mod_nss renegotiate
  [7/22]: disabling mod_nss OCSP
  [8/22]: adding URL rewriting rules
  [9/22]: configuring httpd
  [10/22]: setting up httpd keytab
  [11/22]: configuring Gssproxy
  [12/22]: setting up ssl
  [13/22]: configure certmonger for renewals
  [14/22]: importing CA certificates from LDAP
  [15/22]: publish CA cert
  [16/22]: clean up any existing httpd ccaches
  [17/22]: configuring SELinux for httpd
  [18/22]: create KDC proxy config
  [19/22]: enable KDC proxy
  [20/22]: starting httpd
  [21/22]: configuring httpd to start on boot
  [22/22]: enabling oddjobd
Done configuring the web interface (httpd).
Configuring Kerberos KDC (krb5kdc)
  [1/1]: installing X509 Certificate for PKINIT
Done configuring Kerberos KDC (krb5kdc).
Applying LDAP updates
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/10]: stopping directory server
  [2/10]: saving configuration
  [3/10]: disabling listeners
  [4/10]: enabling DS global lock
  [5/10]: disabling Schema Compat
  [6/10]: starting directory server
  [7/10]: upgrading server
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.
Restarting the KDC
Configuring DNS (named)
  [1/12]: generating rndc key file
  [2/12]: adding DNS container
  [3/12]: setting up our zone
  [4/12]: setting up reverse zone
  [5/12]: setting up our own record
  [6/12]: setting up records for other masters
  [7/12]: adding NS record to the zones
  [8/12]: setting up kerberos principal
  [9/12]: setting up named.conf
  [10/12]: setting up server configuration
  [11/12]: configuring named to start on boot
  [12/12]: changing resolv.conf to point to ourselves
Done configuring DNS (named).
Restarting the web server to pick up resolv.conf changes
Configuring DNS key synchronization service (ipa-dnskeysyncd)
  [1/7]: checking status
  [2/7]: setting up bind-dyndb-ldap working directory
  [3/7]: setting up kerberos principal
  [4/7]: setting up SoftHSM
  [5/7]: adding DNSSEC containers
  [6/7]: creating replica keys
  [7/7]: configuring ipa-dnskeysyncd to start on boot
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
Updating DNS system records
Configuring client side components
Using existing certificate '/etc/ipa/ca.crt'.
Client hostname: centos7.dagyah.com
Realm: DAGYAH.COM
DNS Domain: dagyah.com
IPA Server: centos7.dagyah.com
BaseDN: dc=dagyah,dc=com

Skipping synchronizing time with NTP server.
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
trying https://centos7.dagyah.com/ipa/json
[try 1]: Forwarding 'schema' to json server 'https://centos7.dagyah.com/ipa/json'
trying https://centos7.dagyah.com/ipa/session/json
[try 1]: Forwarding 'ping' to json server 'https://centos7.dagyah.com/ipa/session/json'
[try 1]: Forwarding 'ca_is_enabled' to json server 'https://centos7.dagyah.com/ipa/session/json'
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
[try 1]: Forwarding 'host_mod' to json server 'https://centos7.dagyah.com/ipa/session/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring dagyah.com as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

==============================================================================
Setup complete

Next steps:
        1. You must make sure these network ports are open:
                TCP Ports:
                  * 80, 443: HTTP/HTTPS
                  * 389, 636: LDAP/LDAPS
                  * 88, 464: kerberos
                  * 53: bind
                UDP Ports:
                  * 88, 464: kerberos
                  * 53: bind
                  * 123: ntp

        2. You can now obtain a kerberos ticket using the command: 'kinit admin'
           This ticket will allow you to use the IPA tools (e.g., ipa user-add)
           and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password

→成功したくさい。

[centos7]$ netstat -naptl | grep LISTEN
---(略)---
tcp6       0      0 :::749                  :::*                    LISTEN      2358/kadmind
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::80                   :::*                    LISTEN      6030/httpd
tcp6       0      0 :::8080                 :::*                    LISTEN      5116/java
tcp6       0      0 :::464                  :::*                    LISTEN      2358/kadmind
tcp6       0      0 :::53                   :::*                    LISTEN      6202/named-pkcs11
tcp6       0      0 :::22                   :::*                    LISTEN      6350/sshd
tcp6       0      0 ::1:631                 :::*                    LISTEN      1094/cupsd
tcp6       0      0 :::88                   :::*                    LISTEN      5914/krb5kdc
tcp6       0      0 ::1:953                 :::*                    LISTEN      6202/named-pkcs11
tcp6       0      0 ::1:25                  :::*                    LISTEN      1366/master
tcp6       0      0 :::443                  :::*                    LISTEN      6030/httpd
tcp6       0      0 :::8443                 :::*                    LISTEN      5116/java
tcp6       0      0 :::636                  :::*                    LISTEN      5872/ns-slapd
tcp6       0      0 :::389                  :::*                    LISTEN      5872/ns-slapd

tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      5116/java
tcp6       0      0 127.0.0.1:8009          :::*                    LISTEN      5116/java

→すでにデーモンが起動してた

・windowsPCのhostsファイルにcentos7.dagyah.comエントリを追記してブラウザからアクセス

admin/Password123でログイン

[centos7]$ kinit admin
Password for admin@DAGYAH.COM: 
[centos7]$ ipa user-find admin
--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin@DAGYAH.COM
  UID: 640600000
  GID: 640600000
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------

 

 

Apache Airflow その3 インストール(再)のつづき

 

■ドキュメント

下記を読むことから始める

Installation

tutorial

How-to Guides

■基本的な動作確認

【お題】

・どんなことができるのか?

・どのように動作するのか?

・DAGとは何か?

・タスクインスタンスとは何か?workerとも呼ばれるのか?

・DBをpostgresqlに変更

■Airflowパイプラインの作成、テスト、埋め戻し

※上記の⑤参照

1.基本的なパイプライン定義

※DAG定義ファイルはDAGの構造(DAGオブジェクト)をコードとして指定する単なる構成ファイルでDAG定義ファイルが実際のデータ処理を実行するスクリプトそのものではない。

タスク間の相互通信を行うにはXCom機能を使って実現する。

※DAG定義ファイルではまず必要なライブラリをインポートする文から書く。

※デフォルト引数、テンプレートとかを使って効率的に書く

 

2.DAGのインスタンス化

 

3.Jinjaを使ったテンプレート化

 

4.DAGとタスクのドキュメント追加

 

5.依存関係の設定

 

6.テスト

 

7.Backfill(埋め戻し)

※Backfill(埋め戻し)って何?

 

 

 

 

 

 

 

 

 

Apache Airflow その2 初期設定~ログ設定確認のつづき

 
Apache Airflow その1 インストールでは、airflowをrootでインストールしたために、$AIRFLOW_HOMEが~rootになってしまったがこれはイケてないのでairflowユーザでもう一度インストールしてみる。
 

■ドキュメント

tutorial

How-to Guides

 

■インストール(再)準備

[centos7copy]$ ll /usr/bin/python
lrwxrwxrwx. 1 root root 16 Nov  4 18:10 /usr/bin/python -> /usr/bin/python3

[centos7copy]$ su - airflow
Last login: Wed Nov  4 17:42:01 JST 2020 on pts/6
[airflow@centos7copy ~]$ id
uid=54333(airflow) gid=54333(airflow) groups=54333(airflow) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[airflow@centos7copy ~]$ export AIRFLOW_HOME=~/airflow
[airflow@centos7copy ~]$ echo $AIRFLOW_HOME

/home/airflow/airflow

■インストール(再)

[airflow@centos7copy ~]$ pip3.6 install apache-airflow
Collecting apache-airflow
  Using cached https://files.pythonhosted.org/packages/36/07/368cf47f06564d7ffff603ade4c60039ecf3f5b368b75201f4ccb5512d78/apache_airflow-1.10.12-py2.py3-none-any.whl
Requirement already satisfied: jsonschema~=3.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow)
Collecting pendulum==1.4.4 (from apache-airflow)
  Using cached https://files.pythonhosted.org/packages/30/47/02f04abed54918d2a3f1da602a8254247670b2e1a99b4b1f02734a27e71e/pendulum-1.4.4-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied: python-daemon>=2.1.1 in /usr/local/lib/python3.6/site-packages (from apache-airflow)
Requirement already satisfied: zope.deprecation<5.0,>=4.0 in /usr/local/lib/python3.6/site-packages (from apache-airflow)
Requirement already satisfied: graphviz>=0.12 in /usr/local/lib/python3.6/site-packages (from apache-airflow)
----(略)----
Installing collected packages: pytzdata, pendulum, json-merge-patch, psutil, numpy, pandas, setproctitle, apache-airflow
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/commands/install.py", line 365, in run
    strip_file_prefix=options.strip_file_prefix,
  File "/usr/lib/python3.6/site-packages/pip/req/req_set.py", line 789, in install
    **kwargs
  File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 854, in install
    strip_file_prefix=strip_file_prefix
  File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 1069, in move_wheel_files
    strip_file_prefix=strip_file_prefix,
  File "/usr/lib/python3.6/site-packages/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/lib/python3.6/site-packages/pip/wheel.py", line 316, in clobber
    ensure_dir(destdir)
  File "/usr/lib/python3.6/site-packages/pip/utils/__init__.py", line 83, in ensure_dir
    os.makedirs(path)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/site-packages/pytzdata-2020.1.dist-info'

なんで?

なんかわかんないけど/usr/bin/pythonのシンボリックリンクを元に戻す

[centos7copy]$ rm /usr/bin/python
rm: remove symbolic link ‘/usr/bin/python’? y
[centos7copy]$ ln -s /usr/bin/python2 /usr/bin/python

もういっかい

[airflow@centos7copy ~]$ pip3.6 install apache-airflow

----(略)----

   creating /usr/local/lib/python3.6/site-packages/json_merge_patch
    error: could not create '/usr/local/lib/python3.6/site-packages/json_merge_patch': Permission denied
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-3m0_vqfz/json-merge-patch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-cfbv5qqv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-3m0_vqfz/json-merge-patch/

[centos7copy]$ pip3.6 list | grep json
----(略)----
json-merge-patch (0.2)
jsonschema (3.2.0)
python-json-logger (2.0.1)

[centos7copy]$ pip3.6 show json-merge-patch
Name: json-merge-patch
Version: 0.2
Summary: JSON Merge Patch library (https://tools.ietf.org/html/rfc7386)
Home-page: https://github.com/open-contracting/json-merge-patch
Author: David Raznick
Author-email: mr.raznick@gmail.com
License: BSD
Location: /root/.local/lib/python3.6/site-packages
Requires:

一次的に書き込み権を/usr/local/lib/python3.6/site-packagesに付与w

[centos7copy]$ chmod o+w /usr/local/lib/python3.6/site-packages
[centos7copy]$ ls -ld /usr/local/lib/python3.6/site-packages
drwxr-xrwx. 199 root root 12288 Nov  4 07:50 /usr/local/lib/python3.6/site-packages

[centos7copy]$ chmod o+w /usr/local/lib64/python3.6/site-packages
[centos7copy]$ ls -ld /usr/local/lib64/python3.6/site-packages

drwxr-xrwx. 48 root root 4096 Nov  4 05:57 /usr/local/lib64/python3.6/site-packages

[centos7copy]$ chmod o+w /usr/local/bin/
[centos7copy]$ ls -ld /usr/local/bin/

drwxr-xrwx. 3 root root 4096 Nov  4 05:57 /usr/local/bin/

※インストール後必ず元に戻す!

もういっかい

[airflow@centos7copy ~]$ pip3.6 install apache-airflow
Collecting apache-airflow

----(略)----

Installing collected packages: psutil, numpy, pandas, apache-airflow
  Running setup.py install for psutil ... done
Successfully installed apache-airflow-1.10.12 numpy-1.19.4 pandas-1.1.4 psutil-5.7.3

→成功した臭い

[airflow@centos7copy ~]$ ls -lA ~airflow/
total 32
-rw-------. 1 airflow airflow  746 Nov  4 18:18 .bash_history
-rw-r--r--. 1 airflow airflow   18 Apr  1  2020 .bash_logout
-rw-r--r--. 1 airflow airflow  193 Apr  1  2020 .bash_profile
-rw-r--r--. 1 airflow airflow  231 Apr  1  2020 .bashrc
drwxrwxr-x. 4 airflow airflow 4096 Nov  4 06:03 .cache
drwxrwxr-x. 3 airflow airflow 4096 Nov  4 06:02 .config
-rw-r--r--. 1 airflow airflow  172 Apr  1  2020 .kshrc
drwxr-xr-x. 4 airflow airflow 4096 Nov  1 23:55 .mozilla

→まだairflowディレクトリは居ない

[airflow@centos7copy ~]$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/airflow/.local/bin:/home/airflow/bin

[airflow@centos7copy ~]$ ll /home/airflow/.local/bin /home/airflow/bin
ls: cannot access /home/airflow/.local/bin: No such file or directory
ls: cannot access /home/airflow/bin: No such file or directory

 

あとしまつ

[centos7copy]$ chmod o-w /usr/local/lib/python3.6/site-packages /usr/local/lib64/python3.6/site-packages /usr/local/bin
[centos7copy]$ ls -ld /usr/local/lib/python3.6/site-packages /usr/local/lib64/python3.6/site-packages /usr/local/bin

drwxr-xr-x.   3 root root  4096 Nov  4 18:50 /usr/local/bin
drwxr-xr-x.  58 root root  4096 Nov  4 18:50 /usr/local/lib64/python3.6/site-packages
drwxr-xr-x. 206 root root 12288 Nov  4 18:50 /usr/local/lib/python3.6/site-packages

[centos7copy]$ rm -f /usr/bin/python
[centos7copy]$ ln -s /usr/bin/python3 /usr/bin/python

 

■初期化

[airflow@centos7copy ~]$ pip3.6 list | grep airflow
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
apache-airflow (1.10.12)
[airflow@centos7copy ~]$ pip3.6 show apache-airflow
Name: apache-airflow
Version: 1.10.12
Summary: Programmatically author, schedule and monitor data pipelines
Home-page: http://airflow.apache.org/
Author: Apache Software Foundation
Author-email: dev@airflow.apache.org
License: Apache License 2.0
Location: /usr/local/lib/python3.6/site-packages
Requires: attrs, funcsigs, flask-swagger, zope.deprecation, thrift, typing-extensions, setproctitle, future, croniter, markdown, sqlalchemy-jsonfield, tzlocal, iso8601, graphviz, jinja2, python-dateutil, flask-login, flask, colorlog, python-nvd3, cached-property, gunicorn, jsonschema, lazy-object-proxy, python-daemon, flask-appbuilder, werkzeug, tenacity, pendulum, flask-caching, tabulate, argcomplete, unicodecsv, alembic, dill, flask-admin, email-validator, json-merge-patch, psutil, cattrs, sqlalchemy, pandas, configparser, pygments, python-slugify, requests, flask-wtf

[airflow@centos7copy ~]$ ls -ld /usr/local/lib/python3.6/site-packages/airflow/
drwxrwxr-x. 27 airflow airflow 4096 Nov  4 18:50 /usr/local/lib/python3.6/site-packages/airflow/
[airflow@centos7copy ~]$ ls -ld /usr/local/lib/python3.6/site-packages/apache_airflow-1.10.12.dist-info/
drwxrwxr-x. 2 airflow airflow 4096 Nov  4 18:50 /usr/local/lib/python3.6/site-packages/apache_airflow-1.10.12.dist-info/

[airflow@centos7copy ~]$ ll /usr/local/lib/python3.6/site-packages/airflow/
total 200
-rw-rw-r--.  1 airflow airflow  2237 Nov  4 18:50 alembic.ini
drwxrwxr-x.  6 airflow airflow  4096 Nov  4 18:50 api
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 bin
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 config_templates
-rw-rw-r--.  1 airflow airflow 31180 Nov  4 18:50 configuration.py
drwxrwxr-x. 12 airflow airflow  4096 Nov  4 18:50 contrib
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 dag
-rw-rw-r--.  1 airflow airflow  2646 Nov  4 18:50 default_login.py
drwxrwxr-x.  4 airflow airflow  4096 Nov  4 18:50 example_dags
-rw-rw-r--.  1 airflow airflow  5236 Nov  4 18:50 exceptions.py
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 executors
-rw-rw-r--.  1 airflow airflow    57 Nov  4 18:50 git_version
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 hooks
-rw-rw-r--.  1 airflow airflow  3470 Nov  4 18:50 __init__.py
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 jobs
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 kubernetes
drwxrwxr-x.  4 airflow airflow  4096 Nov  4 18:50 lineage
-rw-rw-r--.  1 airflow airflow  3833 Nov  4 18:50 logging_config.py
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 macros
drwxrwxr-x.  4 airflow airflow  4096 Nov  4 18:50 migrations
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 models
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 operators
-rw-rw-r--.  1 airflow airflow  8165 Nov  4 18:50 plugins_manager.py
drwxrwxr-x.  2 airflow airflow  4096 Nov  4 18:50 __pycache__
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 secrets
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 security
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 sensors
-rw-rw-r--.  1 airflow airflow  5410 Nov  4 18:50 sentry.py
drwxrwxr-x.  3 airflow airflow  4096 Nov  4 18:50 serialization
-rw-rw-r--.  1 airflow airflow 15464 Nov  4 18:50 settings.py
drwxrwxr-x.  4 airflow airflow  4096 Nov  4 18:50 task
drwxrwxr-x.  4 airflow airflow  4096 Nov  4 18:50 ti_deps
-rw-rw-r--.  1 airflow airflow  1207 Nov  4 18:50 typing_compat.py
drwxrwxr-x.  4 airflow airflow  4096 Nov  4 18:50 utils
-rw-rw-r--.  1 airflow airflow   834 Nov  4 18:50 version.py
drwxrwxr-x.  6 airflow airflow  4096 Nov  4 18:50 www
drwxrwxr-x.  6 airflow airflow  4096 Nov  4 18:50 www_rbac

[airflow@centos7copy ~]$ ll /usr/local/lib/python3.6/site-packages/airflow/bin
total 120
-rwxrwxr-x. 1 airflow airflow   1305 Nov  4 18:50 airflow
-rw-rw-r--. 1 airflow airflow 108769 Nov  4 18:50 cli.py
-rw-rw-r--. 1 airflow airflow    811 Nov  4 18:50 __init__.py
drwxrwxr-x. 2 airflow airflow   4096 Nov  4 18:50 __pycache__

 

・初期化実行

[airflow@centos7copy ~]$ /usr/local/lib/python3.6/site-packages/airflow/bin/airflow initdb
DB: sqlite:////home/airflow/airflow/airflow.db
[2020-11-04 19:09:49,085] {db.py:378} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
/usr/local/lib/python3.6/site-packages/alembic/ddl/sqlite.py:44: UserWarning: Skipping unsupported ALTER for creation of implicit constraintPlease refer to the batch mode feature which allows for SQLite migrations using a copy-and-move strategy.
  "Skipping unsupported ALTER for "
INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance
INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log
INFO  [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun
INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration
INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config
INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user
INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end
INFO  [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss
INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table
INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table
INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index
INFO  [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table
INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices
INFO  [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance
INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table
INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
INFO  [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary
INFO  [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff, add ti job_id index
INFO  [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types)
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4, kubernetes_resource_checkpointing
INFO  [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24, kubernetes_resource_checkpointing
INFO  [alembic.runtime.migration] Running upgrade 27c6a30d7c24 -> 86770d1215c0, add kubernetes scheduler uniqueness
INFO  [alembic.runtime.migration] Running upgrade 86770d1215c0, 0e2a74e0fc9f -> 05f30312d566, merge heads
INFO  [alembic.runtime.migration] Running upgrade 05f30312d566 -> f23433877c24, fix mysql not null constraint
INFO  [alembic.runtime.migration] Running upgrade f23433877c24 -> 856955da8476, fix sqlite foreign key
INFO  [alembic.runtime.migration] Running upgrade 856955da8476 -> 9635ae0956e7, index-faskfail
INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> dd25f486b8ea, add idx_log_dag
INFO  [alembic.runtime.migration] Running upgrade dd25f486b8ea -> bf00311e1990, add index to taskinstance
INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> 0a2a5b66e19d, add task_reschedule table
INFO  [alembic.runtime.migration] Running upgrade 0a2a5b66e19d, bf00311e1990 -> 03bc53e68815, merge_heads_2
INFO  [alembic.runtime.migration] Running upgrade 03bc53e68815 -> 41f5f12752f8, add superuser field
INFO  [alembic.runtime.migration] Running upgrade 41f5f12752f8 -> c8ffec048a3b, add fields to dag
INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> dd4ecb8fbee3, Add schedule interval to dag
INFO  [alembic.runtime.migration] Running upgrade dd4ecb8fbee3 -> 939bb1e647c8, task reschedule fk on cascade delete
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 6e96a59344a4, Make TaskInstance.pool not nullable
INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> d38e04c12aa2, add serialized_dag table
Revision ID: d38e04c12aa2
Revises: 6e96a59344a4
Create Date: 2019-08-01 14:39:35.616417
INFO  [alembic.runtime.migration] Running upgrade d38e04c12aa2 -> b3b105409875, add root_dag_id to DAG
INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 74effc47d867, change datetime to datetime2(6) on MSSQL tables
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> a56c9515abdc, Remove dag_stat table
INFO  [alembic.runtime.migration] Running upgrade a56c9515abdc, 004c1210f153, 74effc47d867, b3b105409875 -> 08364691d074, Merge the four heads back together
INFO  [alembic.runtime.migration] Running upgrade 08364691d074 -> fe461863935f, increase_length_for_connection_password
INFO  [alembic.runtime.migration] Running upgrade fe461863935f -> 7939bcff74ba, Add DagTags table
INFO  [alembic.runtime.migration] Running upgrade 7939bcff74ba -> a4c2fd67d16b, add pool_slots field to task_instance
INFO  [alembic.runtime.migration] Running upgrade a4c2fd67d16b -> 852ae6c715af, Add RenderedTaskInstanceFields table
INFO  [alembic.runtime.migration] Running upgrade 852ae6c715af -> 952da73b5eff, add dag_code table
INFO  [alembic.runtime.migration] Running upgrade 952da73b5eff -> a66efa278eea, Add Precision to execution_date in RenderedTaskInstanceFields table
INFO  [alembic.runtime.migration] Running upgrade a66efa278eea -> da3f683c3a5a, Add dag_hash Column to serialized_dag table
Done.

[airflow@centos7copy ~]$ ls -lA ~airflow
total 36
drwxrwxr-x. 3 airflow airflow 4096 Nov  4 19:09 airflow
-rw-------. 1 airflow airflow  746 Nov  4 18:18 .bash_history
-rw-r--r--. 1 airflow airflow   18 Apr  1  2020 .bash_logout
-rw-r--r--. 1 airflow airflow  193 Apr  1  2020 .bash_profile
-rw-r--r--. 1 airflow airflow  231 Apr  1  2020 .bashrc
drwxrwxr-x. 4 airflow airflow 4096 Nov  4 06:03 .cache
drwxrwxr-x. 3 airflow airflow 4096 Nov  4 06:02 .config
-rw-r--r--. 1 airflow airflow  172 Apr  1  2020 .kshrc
drwxr-xr-x. 4 airflow airflow 4096 Nov  1 23:55 .mozilla

→airflowディレクトリが作られた

[airflow@centos7copy ~]$ ls -lA ~airflow/airflow/
total 140
-rw-rw-r--. 1 airflow airflow 38667 Nov  4 19:09 airflow.cfg
-rw-r--r--. 1 airflow airflow 88064 Nov  4 19:09 airflow.db
drwxrwxr-x. 3 airflow airflow  4096 Nov  4 19:09 logs
-rw-rw-r--. 1 airflow airflow  2573 Nov  4 19:09 unittests.cfg

[airflow@centos7copy ~]$ ls -lA ~airflow/airflow/logs
total 4
drwxrwxr-x. 3 airflow airflow 4096 Nov  4 19:09 scheduler

[airflow@centos7copy ~]$ ls -lA ~airflow/airflow/logs/scheduler/
total 4
drwxrwxr-x. 2 airflow airflow 4096 Nov  4 19:09 2020-11-04
lrwxrwxrwx. 1 airflow airflow   47 Nov  4 19:09 latest -> /home/airflow/airflow/logs/scheduler/2020-11-04

 

■Web Server起動

[centos7copy]$ netstat -nap | grep 8080;echo $?
1

[airflow@centos7copy ~]$ /usr/local/lib/python3.6/site-packages/airflow/bin/airflow webserver -p 8080 &
[1] 26190
[airflow@centos7copy ~]$   ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-11-04 19:18:34,132] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:18:34,132] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2020-11-04 19:18:34 +0900] [26198] [INFO] Starting gunicorn 20.0.4
[2020-11-04 19:18:34 +0900] [26198] [INFO] Listening at: http://0.0.0.0:8080 (26198)
[2020-11-04 19:18:34 +0900] [26198] [INFO] Using worker: sync
[2020-11-04 19:18:34 +0900] [26202] [INFO] Booting worker with pid: 26202
[2020-11-04 19:18:34 +0900] [26203] [INFO] Booting worker with pid: 26203
[2020-11-04 19:18:35 +0900] [26204] [INFO] Booting worker with pid: 26204
[2020-11-04 19:18:35 +0900] [26205] [INFO] Booting worker with pid: 26205
[2020-11-04 19:18:35,390] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:18:35,391] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
[2020-11-04 19:18:35,396] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:18:35,397] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
[2020-11-04 19:18:35,450] {__init__.py:50} INFO - Using executor SequentialExecutor
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
[2020-11-04 19:18:35,451] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
[2020-11-04 19:18:35,452] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:18:35,453] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)

[airflow@centos7copy ~]$ netstat -nap | grep 8080
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      26198/gunicorn: mas

[centos7copy]$ pstree -p  26190

・スケジューラを起動する前にWeb serverにアクセスしてみるンゴ

 

→スケジューラが居ないって言ってるぷに

「example_bash_operator schedule」DAGをクリック

[airflow@centos7copy airflow]$ ls -lR /home/airflow/airflow/logs
/home/airflow/airflow/logs:
total 4
drwxrwxr-x. 3 airflow airflow 4096 Nov  4 19:09 scheduler

/home/airflow/airflow/logs/scheduler:
total 4
drwxrwxr-x. 2 airflow airflow 4096 Nov  4 19:09 2020-11-04
lrwxrwxrwx. 1 airflow airflow   47 Nov  4 19:09 latest -> /home/airflow/airflow/logs/scheduler/2020-11-04

/home/airflow/airflow/logs/scheduler/2020-11-04:
total 0

→Web Serverのログは$AIRFLOW_HOME/airflow/logsには作られない

ちなみに、Web Serverを起動したターミナルに標準出力が書き散らかされているw

---(略)---

[2020-11-04 19:31:25 +0900] [26198] [INFO] Handling signal: ttin
[2020-11-04 19:31:25 +0900] [27122] [INFO] Booting worker with pid: 27122
[2020-11-04 19:31:25,896] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:31:25,897] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
[2020-11-04 19:31:26 +0900] [26198] [INFO] Handling signal: ttou
[2020-11-04 19:31:26 +0900] [26965] [INFO] Worker exiting (pid: 26965)


[airflow@centos7copy ~]$ jobs
[1]+  Running                 /usr/local/lib/python3.6/site-packages/airflow/bin/airflow webserver -p 8080 &
[airflow@centos7copy ~]$ [2020-11-04 19:31:56 +0900] [26198] [INFO] Handling signal: ttin
[2020-11-04 19:31:56 +0900] [27158] [INFO] Booting worker with pid: 27158
[2020-11-04 19:31:56,790] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:31:56,791] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
[2020-11-04 19:31:57 +0900] [26198] [INFO] Handling signal: ttou
[2020-11-04 19:31:57 +0900] [27002] [INFO] Worker exiting (pid: 27002)

---(略)---

・スケジューラを起動

別のターミナルからスケジューラを起動

[centos7copy]$ su - airflow
Last login: Wed Nov  4 18:11:47 JST 2020 on pts/6
[airflow@centos7copy ~]$ id
uid=54333(airflow) gid=54333(airflow) groups=54333(airflow) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[airflow@centos7copy ~]$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/airflow/.local/bin:/home/airflow/bin

[airflow@centos7copy ~]$ echo $AIRFLOW_HOME

[airflow@centos7copy ~]$ export AIRFLOW_HOME=~/airflow
[airflow@centos7copy ~]$ echo $AIRFLOW_HOME

/home/airflow/airflow

[airflow@centos7copy ~]$ /usr/local/lib/python3.6/site-packages/airflow/bin/airflow scheduler &
[1] 27703
[airflow@centos7copy ~]$   ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-11-04 19:39:07,861] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:39:07,870] {scheduler_job.py:1367} INFO - Starting the scheduler
[2020-11-04 19:39:07,870] {scheduler_job.py:1375} INFO - Running execute loop for -1 seconds
[2020-11-04 19:39:07,870] {scheduler_job.py:1376} INFO - Processing each file at most -1 times
[2020-11-04 19:39:07,870] {scheduler_job.py:1379} INFO - Searching for files in /home/airflow/airflow/dags
[2020-11-04 19:39:07,873] {scheduler_job.py:1381} INFO - There are 25 files in /home/airflow/airflow/dags
[2020-11-04 19:39:07,874] {scheduler_job.py:1438} INFO - Resetting orphaned tasks for active dag runs
[2020-11-04 19:39:07,910] {dag_processing.py:562} INFO - Launched DagFileProcessorManager with pid: 27717
[2020-11-04 19:39:07,916] {settings.py:55} INFO - Configured default timezone <Timezone [UTC]>
[2020-11-04 19:39:07,926] {dag_processing.py:776} WARNING - Because we cannot use more than 1 thread (max_threads = 2) when using sqlite. So we set parallelism to 1.

[airflow@centos7copy airflow]$ pstree -p 27703
python /usr/loc(27703)---airflow schedul(27717)---airflow schedul(27860)

 

Web Serverをブラウザから見る

[airflow@centos7copy airflow]$ ls -lR $AIRFLOW_HOME
/home/airflow/airflow:
total 144
-rw-rw-r--. 1 airflow airflow 38667 Nov  4 19:09 airflow.cfg
-rw-r--r--. 1 airflow airflow 88064 Nov  4 19:43 airflow.db
-rw-r--r--. 1 airflow airflow     6 Nov  4 19:18 airflow-webserver.pid
drwxrwxr-x. 4 airflow airflow  4096 Nov  4 19:39 logs
-rw-rw-r--. 1 airflow airflow  2573 Nov  4 19:09 unittests.cfg

/home/airflow/airflow/logs:
total 8
drwxr-xr-x. 2 airflow airflow 4096 Nov  4 19:39 dag_processor_manager  ←あらわれた
drwxrwxr-x. 3 airflow airflow 4096 Nov  4 19:09 scheduler

/home/airflow/airflow/logs/dag_processor_manager:
total 48
-rw-rw-r--. 1 airflow airflow 43483 Nov  4 19:43 dag_processor_manager.log

/home/airflow/airflow/logs/scheduler:
total 4
drwxrwxr-x. 2 airflow airflow 4096 Nov  4 19:09 2020-11-04
lrwxrwxrwx. 1 airflow airflow   47 Nov  4 19:09 latest -> /home/airflow/airflow/logs/scheduler/2020-11-04

/home/airflow/airflow/logs/scheduler/2020-11-04:
total 0

 

めんどうくさいからこうする。

[airflow@centos7copy airflow]$ PATH="$PATH:/usr/local/lib/python3.6/site-packages/airflow/bin"

・run your first task instance

[airflow@centos7copy airflow]$ airflow run example_bash_operator runme_0 2015-10-31

[2020-11-04 19:53:22,036] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:53:22,036] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
Running %s on host %s <TaskInstance: example_bash_operator.runme_0 2015-10-31T00:00:00+00:00 [None]> centos7copy
[2020-11-04 19:53:30,570] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 19:53:30,571] {dagbag.py:417} INFO - Filling up the DagBag from /usr/local/lib/python3.6/site-packages/airflow/example_dags/example_bash_operator.py
Running %s on host %s <TaskInstance: example_bash_operator.runme_0 2015-10-31T00:00:00+00:00 [None]> centos7copy

[airflow@centos7copy airflow]$ ls -lR $AIRFLOW_HOME/logs
/home/airflow/airflow/logs:
total 12
drwxr-xr-x. 2 airflow airflow 4096 Nov  4 19:39 dag_processor_manager
drwxrwxrwx. 3 airflow airflow 4096 Nov  4 19:53 example_bash_operator
drwxrwxr-x. 3 airflow airflow 4096 Nov  4 19:09 scheduler

/home/airflow/airflow/logs/dag_processor_manager:
total 176
-rw-rw-r--. 1 airflow airflow 174333 Nov  4 19:55 dag_processor_manager.log

/home/airflow/airflow/logs/example_bash_operator:
total 4
drwxrwxrwx. 3 airflow airflow 4096 Nov  4 19:53 runme_0

/home/airflow/airflow/logs/example_bash_operator/runme_0:
total 4
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 19:53 2015-10-31T00:00:00+00:00

/home/airflow/airflow/logs/example_bash_operator/runme_0/2015-10-31T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 3037 Nov  4 19:54 1.log

/home/airflow/airflow/logs/scheduler:
total 4
drwxrwxr-x. 2 airflow airflow 4096 Nov  4 19:09 2020-11-04
lrwxrwxrwx. 1 airflow airflow   47 Nov  4 19:09 latest -> /home/airflow/airflow/logs/scheduler/2020-11-04

/home/airflow/airflow/logs/scheduler/2020-11-04:
total 0

→上記の青字のファイルやディレクトリが新規作成された

 

・run a backfill over 3 days

[airflow@centos7copy airflow]$ airflow backfill example_bash_operator -s 2020-10-31 -e 2020-01-02
[2020-11-04 20:05:43,909] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-11-04 20:05:43,910] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
[2020-11-04 20:05:49,276] {backfill_job.py:759} INFO - No run dates were found for the given dates and dag interval.

→逆行した日付を指定して怒られたw

→なんにもかわらず

 

[airflow@centos7copy airflow]$ airflow backfill example_bash_operator -s 2020-10-31 -e 2020-11-02

まだ処理中の画面

5分くらい経過。結構時間がかかる

 

10分くらいかかった

→3日分の実行は11分40秒で完了した。

 

[airflow@centos7copy ~]$ ls -lR $AIRFLOW_HOME/logs
/home/airflow/airflow/logs:
total 12
drwxr-xr-x. 2 airflow airflow 4096 Nov  4 19:39 dag_processor_manager
drwxrwxrwx. 8 airflow airflow 4096 Nov  4 20:20 example_bash_operator
drwxrwxr-x. 3 airflow airflow 4096 Nov  4 19:09 scheduler

/home/airflow/airflow/logs/dag_processor_manager:
total 540
-rw-rw-r--. 1 airflow airflow 545597 Nov  4 20:30 dag_processor_manager.log

/home/airflow/airflow/logs/example_bash_operator:
total 24
drwxrwxrwx. 5 airflow airflow 4096 Nov  4 20:17 also_run_this
drwxrwxrwx. 5 airflow airflow 4096 Nov  4 20:19 run_after_loop
drwxrwxrwx. 6 airflow airflow 4096 Nov  4 20:12 runme_0
drwxrwxrwx. 5 airflow airflow 4096 Nov  4 20:14 runme_1
drwxrwxrwx. 5 airflow airflow 4096 Nov  4 20:16 runme_2
drwxrwxrwx. 5 airflow airflow 4096 Nov  4 20:21 run_this_last

/home/airflow/airflow/logs/example_bash_operator/also_run_this:
total 12
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:16 2020-10-31T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:17 2020-11-01T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:17 2020-11-02T00:00:00+00:00

/home/airflow/airflow/logs/example_bash_operator/also_run_this/2020-10-31T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2689 Nov  4 20:17 1.log

/home/airflow/airflow/logs/example_bash_operator/also_run_this/2020-11-01T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2689 Nov  4 20:17 1.log

/home/airflow/airflow/logs/example_bash_operator/also_run_this/2020-11-02T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2689 Nov  4 20:18 1.log

/home/airflow/airflow/logs/example_bash_operator/run_after_loop:
total 12
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:18 2020-10-31T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:19 2020-11-01T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:19 2020-11-02T00:00:00+00:00

/home/airflow/airflow/logs/example_bash_operator/run_after_loop/2020-10-31T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2347 Nov  4 20:19 1.log

/home/airflow/airflow/logs/example_bash_operator/run_after_loop/2020-11-01T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2347 Nov  4 20:19 1.log

/home/airflow/airflow/logs/example_bash_operator/run_after_loop/2020-11-02T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2347 Nov  4 20:20 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_0:
total 16
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 19:53 2015-10-31T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:11 2020-10-31T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:11 2020-11-01T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:12 2020-11-02T00:00:00+00:00

/home/airflow/airflow/logs/example_bash_operator/runme_0/2015-10-31T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 3037 Nov  4 19:54 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_0/2020-10-31T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2380 Nov  4 20:11 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_0/2020-11-01T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2380 Nov  4 20:12 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_0/2020-11-02T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2380 Nov  4 20:12 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_1:
total 12
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:13 2020-10-31T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:13 2020-11-01T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:14 2020-11-02T00:00:00+00:00

/home/airflow/airflow/logs/example_bash_operator/runme_1/2020-10-31T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2380 Nov  4 20:13 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_1/2020-11-01T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2382 Nov  4 20:14 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_1/2020-11-02T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2382 Nov  4 20:14 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_2:
total 12
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:14 2020-10-31T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:15 2020-11-01T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:16 2020-11-02T00:00:00+00:00

/home/airflow/airflow/logs/example_bash_operator/runme_2/2020-10-31T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2382 Nov  4 20:15 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_2/2020-11-01T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2382 Nov  4 20:16 1.log

/home/airflow/airflow/logs/example_bash_operator/runme_2/2020-11-02T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 2382 Nov  4 20:16 1.log

/home/airflow/airflow/logs/example_bash_operator/run_this_last:
total 12
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:20 2020-10-31T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:20 2020-11-01T00:00:00+00:00
drwxrwxrwx. 2 airflow airflow 4096 Nov  4 20:21 2020-11-02T00:00:00+00:00

/home/airflow/airflow/logs/example_bash_operator/run_this_last/2020-10-31T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 1833 Nov  4 20:20 1.log

/home/airflow/airflow/logs/example_bash_operator/run_this_last/2020-11-01T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 1833 Nov  4 20:21 1.log

/home/airflow/airflow/logs/example_bash_operator/run_this_last/2020-11-02T00:00:00+00:00:
total 4
-rw-rw-rw-. 1 airflow airflow 1833 Nov  4 20:22 1.log

/home/airflow/airflow/logs/scheduler:
total 4
drwxrwxr-x. 2 airflow airflow 4096 Nov  4 19:09 2020-11-04
lrwxrwxrwx. 1 airflow airflow   47 Nov  4 19:09 latest -> /home/airflow/airflow/logs/scheduler/2020-11-04

/home/airflow/airflow/logs/scheduler/2020-11-04:
total 0

 

【airflow起動の中間まとめ】

※rootであらかじめ下記を実行しておく

ln -s /usr/bin/python3 /usr/bin/python

①su - airflow

②export PATH="$PATH:/usr/local/lib/python3.6/site-packages/airflow/bin"

③export AIRFLOW_HOME=~/airflow

④airflow webserver -p 8090 &

 ※ポート番号は開いてるポート番号を指定する。

⑤airflow scheduler &